-
Learn Type Variables and Generic Types in Python with Simple Examples
In this post, we will introduce the basics of type variables and generic functions and classes with simple code examples. Using type variables and generic types can make our code more robust and easier to read. Even if we may not have many chances to use them directly in our own code, it’s still very… Continue reading
-
How to Test a Web Application on Your Phone Directly
When developing a web application with Angular, or any other framework, we need to test it on our phones frequently. Even though we can use the “device toolbar” simulator in Chrome for testing, sometimes we still need to use a real phone for testing. For example, when we want to check the keyboard behavior in… Continue reading
-
Some Special Use Cases of Regular Expressions in Python
In this post, we will showcase a selection of special applications for regular expressions in Python, aimed at resolving real-world challenges. As we will see in our examples, we can use word boundaries and lookahead assumptions to achieve more accurate matching. A small trick will also be introduced for how to create consistent match strings… Continue reading
-
Use pipdeptree to Check Python Package Dependencies and Create a Clean requirements.txt
Learn a handy tool to check package dependencies in Python Writing a requirements.txt file from scratch for your application can be cumbersome because you need to sort out what packages need to be installed and what are their dependencies. Worse still, it’s tedious to resolve conflicting dependencies, especially on legacy systems when an older version of… Continue reading
-
How to Use Zyte Smart Proxy with Scrapy and Splash
Learn how to scrape JavaScript webpages with a smart proxy In this post, we will introduce how to use the Zyte Smart proxy with Splash which is integrated with the Scrapy web scraping framework. We will learn how to set up the Zyte smart proxy, how to use it with Splash, and how to set up… Continue reading
-
How to Install Private Python Packages in Artifact Registry When Building Docker Images with Cloud Build
Learn to build and publish both Python packages and Docker images with Cloud Build In previous posts, we introduced how to store Python packages and Docker images in Google Artifact Registry, respectively. In this post, we will try to connect these two techniques and introduce how to install private Python packages in Artifact Registry when building… Continue reading
-
How to Migrate from Google Container Registry to Artifact Registry
Learn a simple way to perform smooth transitions Container Registry has been deprecated since May 15, 2023. In order to get the latest updates, features, and support for the services, we should transition from Container Registry to Artifact Registry. In this post, we will first introduce the differences between Container Registry and Artifact Registry and… Continue reading
-
How to Start a Splash Microservice for Scraping JavaScript Webpages in Google Cloud Run
Learn a simple way to build a production-ready Splash server in the Cloud In previous posts, we introduced how to start Splash locally using Docker. In production, we would want to put the Splash server somewhere all our scraping projects can access. Since Splash is Dockerized, a simple solution is to start a Splash microservice in… Continue reading
-
How to Store Docker Images in Google Artifact Registry
Learn a new way to store Docker images on the Google Cloud Platform We used to store Docker images in the Container Registry on the Google Cloud Platform (GCP). However, it’s deprecated now and we should instead store Docker images in the Artifact Registry which is a central location for storing and managing “build artifacts” such… Continue reading
-
How to Use Splash with Proxies for Scraping JavaScript Webpages
Learn different ways to use proxies for web scraping with Splash In web scraping, we often need to use proxies because of IP blocking or geolocation-sensitive data which means the scraped data can change depending on the geolocations of the IPs making the requests. It is straightforward to specify proxies for regular web scraping. However, when… Continue reading