Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
D
documentation
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 0
    • Issues 0
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 0
    • Merge Requests 0
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • dbnomics-fetchers
  • documentation
  • Wiki
  • code optimization

Last edited by Christophe Benz Jan 30, 2019
Page history

code optimization

Articles

  • https://pythonfiles.wordpress.com/2017/06/01/hunting-performance-in-python-code-part-3/

Profiling CPU

With cProfile

Collect data:

python -m cProfile -o out.cprof script.py args...

Use snakeviz to visualize collected data:

pip install snakeviz

snakeviz out.cprof

Alternatives to snakeviz:

pip install pyprof2calltree
apt install qcachegrind

pyprof2calltree -k -i out.cprof

With pprofile

https://github.com/vpelletier/pprofile

With line_profiler

https://github.com/rkern/line_profiler

Profiling memory

With memory_profiler

Global memory usage

Note: use the --python python3 option as indicated in this issue.

mprof run --python python3 convert.py source-data json-data
mprof plot

Optionally decorate some functions with @profile, without importing the decorator, as explained in the README. You'll see the memory usage of those functions explicitly on the chart.

Line by line

Decorate the functions you want to profile with @profile and run with:

python -m memory_profiler convert.py source-data json-data

Misc

  • https://github.com/nvdv/vprof
Clone repository
  • Code style
  • Git and Gitlab workflow
  • acceptance criteria
    • fetchers
  • ci jobs and runners
  • code optimization
  • dev tools
  • e mails
  • failure handling procedures
  • Home
  • librairies
  • maintaining fetchers
  • monitoring
  • presentation
  • production configuration
  • publishing python packages
View All Pages