Tutorials
I was working on optimizing some code that contained a series of loops. I began my analysis by running a few different versions of the program and timing each execution. The results were enlightening! I decided to share the approach with my team along with an essay on the topic by Guido Van Rossem.
General Rules of Thumb by Which to Develop Loops
- Never optimize before you have proven a speed bottleneck exists. Then, only optimize the innermost loop. If you have a bunch of loops, consider breaking the function apart.
- Python incurs significant lookup charges for bytecode instructions and variable lookup, it rarely improves code to add extra conditionals/checks to a function to save a little work e.g. handling known scenarios with unpythonic case-like statements.
- Use intrinsic operations. An implied loop in
map()is faster than an explicit for loop; An explicit for loop is faster than a while loop with a loop counter.
- Use intrinsic operations. An implied loop in
I recently started working on a workflow for picking up files from S3, processing them, and writing the results to another S3 location. This is a common pattern in data processing pipelines and our team wanted to see whether we could do it using AWS serverless services. We were able to get it running via Lambda functions and event triggers published to an AWS EventHub. The entire workflow was fairly easy to stand up once we grasped how the various services worked together.
I started the process of moving my personal blog over a year ago, when the global pandemic brought on by the COVID-19 virus sent my local area into lock down. The reasons for doing so were simple enough, my web hosting bill had grown north of $200 dollars a month for my personal blog. Don’t get me wrong, Wordpress is great! I just wanted to get my blog onto something more appropriate for the audience (read, pretty small).
Sometimes the solution to a problem is so obvious, it takes a while to figure it out. I recently stumbled on such a problem when trying to configure a set of Object Relational Mappings (ORM) to support an application with the same set of table objects across different schemas in Postgres. Developing an ORM to support this pattern, a multi-tenant database model, proved challenging because of where I started. Below, I will detail the correct way to support the multi-tenant pattern as well as various approaches I came across and why they should not be used.
I have been waiting for a project that would allow me to dig into the Python’s asyncio library. Recently, such a project presented itself. I was tasked with hitting a rate limited REST API with just under 4 million requests. My first attempt was simple. Gather and build a block of search queries, POST each one to the API, process the results, and finally insert them in a database. Here is what the code looked like: