Scrapy Settings:

If you’re looking to uniquely customize your scraper, then you’ll want to learn the tips and tricks of how to do this without a hassle. Using Scrapy settings, you can conveniently customize the crawling settings of your crawler. That’s not all, scrapy also allow you to customize other items like core mechanism, pipelines, and spiders. You’ll typically find a settings.py file in your project directory that allows you to easily customize your scraper’s settings.

Custom commands or what many have come to know as management commands or utility commands are one of the essential features that are provided by python frameworks. These commands are quite useful when performing tasks that requires a lot of different methods to complete the task.

Concurrency in Python is no doubt a complex topic and one that is hard to understand. More so, it also doesn’t help that there are multiple ways to produce concurrent programs. For a lot of people, they have to deal with lots of thoughts including asking questions like,
Should I spin up multiple threads?
Use multiple processes?
Use asynchronous programming?
Here is the thing, you should use async IO when you can and use threading when you must.