Python Bytes

Sponsored by Datadog: pythonbytes.fm/datadog Michael #1: Guido retiresGuido van Rossum has left DropBox and retired (post)Let’s all take a moment to say thank you.I wonder what will come next in terms of creative projectsSome comments from community members (see Twitter thread)Brian #2: SeleniumBaseAutomated UI Testing with Selenium WebDriver and pytest.Very expressive and intuitive automation library built on top of Selenium WebDriver. method overviewvery readable (this is a workflow test, but still, quite readable):from seleniumbase import BaseCase class MyTestClass(BaseCase): def test_basic(self): self.open("https://xkcd.com/353/") self.assert_title("xkcd: Python") self.assert_element('img[alt="Python"]') self.click('a[rel="license"]') self.assert_text("free to copy and reuse") self.go_back() self.click("link=About") self.assert_text("xkcd.com", "h2") self.open("https://store.xkcd.com/collections/everything") self.update_text("input.search-input", "xkcd book\n") self.assert_exact_text("xkcd: volume 0", "h3")includes plugins for including screenshots in test results.supports major CI systemssome cool features that I didn’t expectuser onboarding demosassisted QA (partially automated with manual questions)support for selenium gridlogs of command line options, including headlessMichael #3: Reimplementing a Solaris command in Python gained 17x performance improvement from CPostmortem by Darren MoffatIs Python slow?A result of fixing a memory allocation issue in the /usr/bin/listusers commandDecided to investigate if this ancient C code could be improved by conversion to Python.The C code was largely untouched since 1988 and was around 800 lines long, it was written in an era when the number of users was fairly small and probably existed in the local files /etc/passwd or a smallish NIS server.It turns out that the algorithm to implement the listusers is basically some simple set manipulation.Rewrite of listusers in Python 3 turned out to be roughly a 10th of the number of lines of codeBut Python would be slower right ? Turns out it isn't and in fact for some of my datasets (that had over 100,000 users in them) it was 17 times faster.A few of the comments asked about the availability of the Python version. The listusers command in Oracle Solaris 11.4 SRU 9 and higher. Since we ship the /usr/bin/listusers file as the Python code you can see it by just viewing the file in an editor. Note though that is not open source and is covered by the Oracle Solaris licenses.Brian #4: 20 useful Python tips and tricks you should knowI admit it, I’m capable of getting link-baited by the occasional listicle.Some great stuff, especially for people coming from other languages.Chained assignment: x = y = z = 2Chained comparison: 2 < x <= 82 < x > 40 < x < 4 < y < 16Multiple assignment: x, y, z = 2, 4, 8More Advanced Multiple Assignment: x, *y, z = 2, 4, 8, 16I’ve been using the * for unpacking a lot, especially with *_Merge dictionaries: z = {**x, **y}Join strings: '_'.join(['u', 'v', 'w'])using list(set(something)) to remove duplicates.aggregate elements. using zip to element-wise combine two or more iterables. >>> x = [1, 2, 3] >>> y = ['a', 'b', 'c'] >>> zip(x, y) [(1, 'a'), (2, 'b'), (3, 'c')]and then some other weird stuff that I don’t find that useful.Michael #5: Complexity Waterfallvia Ahrem AhreffHeavy use of wemake-python-styleguideCode smells!Use your refactoring tools and write tests.Automation enable an opportunity of “Continuous Refactoring” and “Architecture on Demand” development styles.Brian #6: PlynthPlynth is a GUI framework for building cross-platform desktop applications with HTML, CSS and Python.Plynth has integrated the standard CPython implementation with Chromium's rendering engine. You can run your python scripts seamlessly with HTML/CSS instead of using Javascript with modules from pipPlynth uses Chromium/Electron for its rendering. With Plynth, every Javascript library turns into a Python module.Not open source. But free for individuals, including commercial use and education.A bunch of tutorial videos that are not difficult to follow, and not long, but… not really obvious code either. Python 3.6 and 3.7 development kits availableExtras:Michael: Google Is Uncovering Hundreds Of Race Conditions Within The Linux KernelJoke:Q: What's a web developer's favorite tea?A: URL grayvia Aideen Barry

Sponsored by Datadog: pythonbytes.fm/datadog Special guest: Bob BelderbosBrian #1: Lesser Known Coding FontsInteresting examination of some coding fonts.Link to a great talk called Cracking the Code, by Jonathan David Ross, about coding fonts and Input.I’m trying out Input Mono right now, and quite like it.Fira code: https://github.com/tonsky/FiraCode Bob #2: Django Admin HandbookAs a Django developer knowing the admin is pretty important.Free ebook of 40 or so pages, you can consume it in one evening.There are a lot of good tricks, 3 I liked:How to optimize queries in Django admin (override get_queryset)How to export CSV from Django admin (useful for data analysis in Jupyter for example) How to override save behaviour for Django admin (used this to notify users upon publishing a new exercise on our platform)Some more cool ebooks on that site, e.g. Tweetable #Python.Michael #3: Your Guide to the CPython Source CodeLet’s talk about exploring the CPython codeYou’ll want to get the code: git clone https://github.com/python/cpythonCompile the code (Anthony gives lots of steps for macOS, Windows, and Linux)Structure: cpython/ │ ├── Doc ← Source for the documentation ├── Grammar ← The computer-readable language definition ├── Include ← The C header files ├── Lib ← Standard library modules written in Python ├── Mac ← macOS support files ├── Misc ← Miscellaneous files ├── Modules ← Standard Library Modules written in C ├── Objects ← Core types and the object model ├── Parser ← The Python parser source code ├── PC ← Windows build support files ├── PCbuild ← Windows build support files for older Windows versions ├── Programs ← Source code for the python executable and other binaries ├── Python ← The CPython interpreter source code └── Tools ← Standalone tools useful for building or extending PythonSome cool “hidden” goodies. For example, check out Lib/concurrent/futures/process.py, it comes with a cool ascii diagram of the process.Lots more covered, that we don’t have time forThe Python Interpreter ProcessThe CPython Compiler and Execution LoopObjects in CPythonThe CPython Standard LibraryInstalling a custom versionBrian #4: Six Django template tags not often used in tutorialsHere’s a few:{% empty %}, for use in for loops when the array is empty{% lorem \[count\] [method] [random] %} for automatically filling with Lorem Ipsum text.{% verbatim %} … {% endverbatim %}, stop the rendering engine from trying to parse it and replace stuff.https://hipsum.co/Bob #5: Beautiful code snippets with CarbonBeautiful images, great for teaching Python / programming.Used by a lot of developer, nice example I spotted today.Supports typing and drag and drop, just generated this link by dropping a test module onto the canvas!Great to expand Twitter char limit (we use it to generate Python Tip images).Follow the project here, seems they now integrate with Github.Michael #6: Researchers find bug in Python script may have affected hundreds of studiesMore info via Mike Driscoll at Thousands of Scientific Papers May be Invalid Due to Misunderstanding PythonIn a paper published October 8, researchers at the University of Hawaii found that a programming error in a set of Python scripts commonly used for computational analysis of chemistry data returned varying results based on which operating system they were run on.Scientists did not understand that Python’s glob.glob() does not return sorted resultsThrowing doubt on the results of more than 150 published chemistry studies.the researcher were trying to analyze results from an experiment involving cyanobacteria discovered significant variations in results run against the same nuclear magnetic resonance spectroscopy (NMR) data.The scripts, called the "Willoughby-Hoye" scripts after their creators, were found to return correct results on macOS Mavericks and Windows 10. But on macOS Mojave and Ubuntu, the results were off by nearly a full percent.The module depends on the operating system for the order in which the files are returned. And the results of the scripts' calculations are affected by the order in which the files are processed.The fix: A simple list.sort()!Williams said he hopes the paper will get scientists to pay more attention to the computational side of experiments in the future.Extras:Nov 5 is the next Python PDX WestUsing Big Tech ToolsWorking on: PyBites platform: added flake8/ black code formatting, UI enhancements. Michael:Bezos DDoS'd: Amazon Web Services' DNS systems knackered by hours-long cyber-attackPyPI Just Crossed the 200,000 Packages Threshold! (via RP)XKCD Date — via André Jaenisch, Enter https://explainxkcd.com/wiki/index.php/1425:_Tasks and learn, that it was published on 24th Sep 2014.Joke:Q: What did the Network Administrator say when they caught a nasty virus?A: It hurts when IP

Sponsored by DigitalOcean: pythonbytes.fm/digitaloceanMichael #1: Building a Python C Extension ModuleTutorial, learn to use the Python API to write Python C extension modules.AndInvoke C functions from within PythonPass arguments from Python to C and parse them accordinglyRaise exceptions from C code and create custom Python exceptions in CDefine global constants in C and make them accessible in PythonTest, package, and distribute your Python C extension moduleExtending Your Python Programthere may be other lesser-used system calls that are only accessible through CSteps: Writing a Python Interface in CFigure out the arguments (e.g. int fputs(const char *, FILE *) )Implement in C: #include Python.h static PyObject *method_fputs(PyObject *self, PyObject *args) { char *str, *filename = NULL; int bytes_copied = -1; /* Parse arguments */ if(!PyArg_ParseTuple(args, "ss", &str, &filename)) { return NULL; } FILE *fp = fopen(filename, "w"); bytes_copied = fputs(str, fp); fclose(fp); return PyLong_FromLong(bytes_copied); }In line 2, you declare the argument types you wish to receive from your Python codeline 6, then you’ll see that PyArg_ParseTuple() copies into the char*’sWrite regular C code (fopen, fputs)Return: PyLong_FromLong() creates a PyLongObject, which represents an integer object in Python. a few extra functions that are necessarywrite definitions of your module and the methods it containsBefore you can import your new module, you first need to build it. You can do this by using the Python package distutils.Brian #2: What’s New in Python 3.8 - docs.python.orgWe’ve already talked about the big hitters:assignment expressions, (the walrus operator)positional only parameters, (the / in the param list)f-strings support = for self-documenting expressions and debuggingThere are a few more goodies I wanted to quickly mention:More async: python -m asyncio launches a native async REPLMore helpful warnings and messages whenusing is and is not to compare strings and integers and other types intended to be compared with == and !=Missing the comma in stuff like [(1,2) (3,4)]. Happens all the time with parametrized testingyou can do iterable unpacking in a yield or return statementx = (1, 2, 3)a, *b = xreturn a, *b <- this used to be a syntax erroryou had to do return (a, *b)New module importlib.metadata lets you access things like version numbers or dependent library required version numbers, and cool stuff like that.quite a few more goodies. I run through all my favorites on testandcode.com/91Michael #3: UK National Cyber Security Centre (NCSC) is warning developers of the risks of sticking with Python 2.7, particularly for library writersNCSC likens companies continuing to use Python 2 past its EOL to tempting another WannaCry or Equifax incident.Equifax details: a vulnerability, dubbed CVE-2017-5638, was discovered in Apache Struts, an open source development framework for creating enterprise Java applications that Equifax, along with thousands of other websites, uses…Quote: "If you're still using 2.x, it's time to port your code to Python 3," the NCSC said. "If you continue to use unsupported modules, you are risking the security of your organisation and data, as vulnerabilities will sooner or later appear which nobody is fixing."Moreover: "If you maintain a library that other developers depend on, you may be preventing them from updating to 3," the agency added. "By holding other developers back, you are indirectly and likely unintentionally increasing the security risks of others.”"If migrating your code base to Python 3 is not possible, another option is to pay a commercial company to support Python 2 for you," the NCSC said.NCSC: If you don't migrate, you should expect security incidentsPython's popularity makes updating code imperative: The reason the NCSC is warning companies about Python 2's impending EOL is because of the language's success.Brian #4: Pythonic NewsSebastian A. Steins“A Hacker News lookalike written in Python/Django”“ powering https://news.python.sc"Cool that it’s open source, and on githubWas submitted to us by Sebastian, and a few others too, so there is excitement.It’s like 6 days old and has 153 stars on github, 4 contributors, 18 forks.Fun. Michael #5: Deep Learning Workstations, Servers, Laptops, and GPU CloudGPU-accelerated with TensorFlow, PyTorch, Keras, and more pre-installed. Just plug in and start training. Save up to 90% by moving off your current cloud and choosing Lambda.They offer:TensorBook: GPU Laptop for $2,934Lambda Quad: 4x GPU Workstation for $21,108 (yikes!)All in: Lambda Hyperplane: 8x Tesla V100 Server, starting at $114,274But compare to: AWS EC2: p3.8xlarge @ $12.24 per Hour => $8,935 / monthBrian #6: Auto formatters for PythonA comparison of autopep8, yapf, and blackAuto formatters are super helpful for teams. They shut down the unproductive arguments over style and make code reviews way more pleasant. People can focus on content, not being the style police.We love black. But it might be a bit over the top for some people. Here are a couple of other alternatives.autopep8 - mostly focuses on PEP8“autopep8 automatically formats Python code to conform to the PEP 8 style guide. It uses the pycodestyle utility to determine what parts of the code needs to be formatted. autopep8 is capable of fixing most of the formatting issues that can be reported by pycodestyle.”black - does moredoesn’t have many options, but you can alter line length, can turn of string quote normalization, and you can limit or focus the files it sees.does a cool “check that the reformatted code still produces a valid AST that is equivalent to the original.” but you can turn that off with --fastyapf - way more customizable. Great if you want to auto format to a custom style guide.“The ultimate goal is that the code YAPF produces is as good as the code that a programmer would write if they were following the style guide. It takes away some of the drudgery of maintaining your code.”Article is cool in that it shows some sample code and how it’s changed by the different formatters.Extras:Michael:New courses comingFinancial Aid Launches for PyCon US 2020!Joke:American Py SongFrom Eric Nelson:Math joke. “i is as complex as it gets. jk.”

Sponsored by DigitalOcean: pythonbytes.fm/digitaloceanMichael #1: JPMorgan’s Athena Has 35 Million Lines of Python 2 Code, and Won’t Be Updated to Python 3 in TimeWith 35 million lines of Python code, the Athena trading platform is at the core of JPMorgan's business operations. A late start to migrating to Python 3 could create a security risk.Athena platform is used internally at JPMorgan for pricing, trading, risk management, and analytics, with tools for data science and machine learning. This extensive feature set utilizes over 150,000 Python modules, over 500 open source packages, and 35 million lines of Python code contributed by over 1,500 developers, according to data presented by Misha Tselman, executive director at J.P. Morgan Chase in a talk at PyData 2017.And JPMorgan is going to miss the deadlineRoadmap puts "most strategic components" compatible with Python 3 by the end of Q1 2020JPMorgan uses Continuous Delivery, with 10,000 to 15,000 production changes per week"If you maintain a library that other developers depend on," the post states, "you may be preventing them from updating to 3. By holding other developers back, you are indirectly and likely unintentionally increasing the security risks of others," adding that developers who do not publish code publicly should "consider your colleagues who may also be using your code internally."Brian #2: organizesuggested by Ariel Barkana Python based file management automation toolconfiguration is via a yml filecommand line tool to organize your file systemexamples:move all of your screenshots off of your desktop into a screenshots foldermove old incomplete downloads into trashremove empty files from certain foldersorganize receipts and invoices into date based foldersMichael #3: PEP 589 – TypedDict: Type Hints for Dictionaries With a Fixed Set of KeysAuthor: Jukka LehtosaloSponsor: Guido van RossumStatus: AcceptedVersion: 3.8PEP 484 defines the type Dict[K, V] for uniform dictionaries, where each value has the same type, and arbitrary key values are supported. It doesn't properly support the common pattern where the type of a dictionary value depends on the string value of the key.Core idea: Consider creating a type to validate an arbitrary JSON document with a fixed schemaProposed syntax: from typing import TypedDict class Movie(TypedDict): name: str year: int movie: Movie = {'name': 'Blade Runner', 'year': 1982}Operations on movie can be checked by a static type checker movie['director'] = 'Ridley Scott' # Error: invalid key 'director' movie['year'] = '1982' # Error: invalid value type ("int" expected)Brian #4: gazpachogazpacho is a web scraping library“It replaces requests and BeautifulSoup for most projects. ““gazpacho is small, simple, fast, and consistent.”example of using gazpacho to scrape hockey data for fantasy sports.simple interface, short scripts, really beginner friendlyretrieve with get, parse with Soup.I don’t think it will completely replace the other tools, but for simple get/parse/find operations, it may make for slimmer code.Note, I needed to update certificates to get this to work. see this.Michael #5: How pip install Worksvia PyDistWhat happens when you run pip install [somepackage]?First pip needs to decide which distribution of the package to install.This is more complex for Python than many other languagesThere are 7 different kinds of distributions, but the most common these days are source distributions and binary wheels. A binary wheel is a more complex archive format, which can contain compiled C extension code. Compiling, say, numpy from source takes a long time (~4 minutes on my desktop), and it is hard for package authors to ensure that their source code will compile on other people's machines.Most packages with C extensions will build multiple wheel distributions, and pip needs to decide which if any are suitable for your computer.To find the distributions available, pip requests https://pypi.org/simple/[somepackage], which is a simple HTML page full of links, where the text of the link is the filename of the distribution.To select a distribution, pip first determines which distributions are compatible with your system and implementation of python.binary wheels, it parses the filenames according to PEP 425, extracting the python implementation, application binary interface, and platform.All source distributions are assumed to be compatible, at least at this step in the processOnce pip has a list of compatible distributions, it sorts them by version, chooses the most recent version, and then chooses the "best" distribution for that versionIt prefers binary wheels if there are anyDetermining the dependencies for this distribution is not simple either. For binary wheels, the dependencies are listed in a file called METADATA. But for source distributions the dependencies are effectively whatever gets installed when you execute their setup.py script with the install command. What happens though if one of the distributions pip finds violates the requirements of another? It ignores the requirement and installs idna anyway!Next pip has to actually build and install the package.it needs to determine which library directory to install the package in—the system's, the user's, or a virtualenvs? Controlled by sys.prefix, which in turn is controlled by pip's executable path and the PYTHONPATH and PYTHONHOME environment variables. Finally, it moves the wheel files into the appropriate library directory, and compiles the python source files into bytecode for faster execution.Now your package is installed!Brian #6: daily pandas tricksKevin Markham is sending out one pandas tip or trick per day via twitter.It’s been fun to watch and learn new bits.The link is a sampling of a bunch of them.Here’s just one example: Need to rename all of your columns in the same way? Use a string method: Replace spaces with _: df.columns = df.columns.str.replace(' ', '_') Make lowercase & remove trailing whitespace: df.columns = df.columns.str.lower().str.rstrip()ExtrasMichael:Switched to Adobe AuditionAzure Databricks drops Python 2Better Jupyter in VS Code macOS Catalina (so far so good)Jokes:via Sarcastic PharmacistHard to distinguish hard from easy in programming

Sponsored by DigitalOcean: pythonbytes.fm/digitaloceanBrian #1: Python alternative to DockerMatt LaymanUsing Shiv, from LinkedInMentioned briefly in episode 114Shiv uses zipapp, PEP 441.Execute code directly from a zip file.App code and dependencies can be bundled together.“Having one artifact eliminates the possibility of a bad interaction getting to your production system.”article includes an example of all the steps for packaging a Django app with Gunicorn.includes talking about deployment.Matt includes shoutouts to:Platform as a Service providersManual steps to do it all.DockerCompares the process against Docker and discusses when to choose one over the other.Also an interesting read: Docker is in deep troubleMichael #2: How to support open-source software and stay sanevia Jason Thomas written by Anna Nowogrodzki Releasing lab-built open-source software often involves a mountain of unforeseen work for the developers.Article opens: “On 10 April, astrophysicists announced that they had captured the first ever image of a black hole. This was exhilarating news, but none of the giddy headlines mentioned that the image would have been impossible without open-source software.”The image was created using Matplotlib, a Python library for graphing data, as well as other components of the open-source Python ecosystem. Just five days later, the US National Science Foundation (NSF) rejected a grant proposal to support that ecosystem, saying that the software lacked sufficient impact.Open-source software is widely acknowledged as crucially important in science, yet it is funded non-sustainably.“It’s sort of the difference between having insurance and having a GoFundMe when their grandma goes to the hospital,” says Anne CarpenterChallengesScientists writing open-source software often lack formal training in software engineering.Yet poorly maintained software can waste time and effort, and hinder reproducibility.If your research group is planning to release open-source software, you can prepare for the support workObsolescence isn’t bad, she adds: knowing when to stop supporting software is an important skill.However long your software will be used for, good software-engineering practices and documentation are essential.These include continuous integration systems (such as TravisCI), version control (Git) and unit testing.To facilitate maintenance, Varoquaux recommends focusing on code readability over peak performance.Brian #3: The Hippocratic LicenseCoraline Ada EhmkeInteresting idea to derive from MIT, but add restrictions.This license adds these restrictions:“The software may not be used by individuals, corporations, governments, or other groups for systems or activities that actively and knowingly endanger, harm, or otherwise threaten the physical, mental, economic, or general well-being of individuals or groups in violation of the United Nations Universal Declaration of Human Rights”I could see others with different restrictions, or this but more.Michael #4: MATLAB vs Python: Why and How to Make the SwitchMATLAB® is widely known as a high-quality environment for any work that involves arrays, matrices, or linear algebra.I personally used it for wavelet-decomposition of real time eye measurements during cognitively intensive human workloads… That toolbox costs $2000 per user.Difference in philosophy: Closed, paid vs. open source.Since Python is available at no cost, a much broader audience can use the code you developAlso, there is GNU Octave is a free and open-source clone of MATLAB apparentlyBrian #5: PyperCard - Easy GUIs for AllNicholas TollerveyCame up on episode 143Also, episode 89 of Test & CodeReally easy to quickly set up a GUI specified by a list of “Card” objects. (different from cards project)Simple examples are choose your own adventure type applications, where one button takes you to another card, and another button, a different card.However, the “next card” could be a Python function that can do anything, as long as it returns a string with the name of the next card.Lots of potential here, especially with input boxes, images, sound, and more.Super fun, but also might have business use.Michae #6: pynodeArticle: Bridging Node.js and Python with PyNode to Predict Home PricesCall python code from node.jsDefine a Python methodIn node: require pynode: const pynode = require('@fridgerator/pynode')Start an interpreter: pynode.startInterpreter()Call the function pynode.call('add', 1, 2, (err, result) => { if (err) return console.log('error : ', err) result === 3 // true })Jokes The "Works on My Machine" Certification Program, get certified!

Sponsored by Datadog: pythonbytes.fm/datadogMichael #1: How to Stand Out in a Python Coding InterviewReal Python, by James TimminsAre tech interviews broken? Well at least we can try to succeed at them anywayYou’ve made it past the phone call with the recruiter, and now it’s time to show that you know how to solve problems with actual code…Interviews aren’t just about solving problems: they’re also about showing that you can write clean production code. This means that you have a deep knowledge of Python’s built-in functionality and libraries.Things to learnUse enumerate() to iterate over both indices and valuesDebug problematic code with breakpoint()Format strings effectively with f-stringsSort lists with custom argumentsUse generators instead of list comprehensions to conserve memoryDefine default values when looking up dictionary keysCount hashable objects with the collections.Counter classUse the standard library to get lists of permutations and combinationsBrian #2: The Python Software Foundation has updated its Code of ConductThere’s now one code of conduct for PSF and PyCon US and other spaces sponsored by the PSFThis includes some regional conferences, such as PyCascades, and some meetup groups, (ears perk up)The docsCode of ConductEnforcement GuidelinesReporting GuidelinesDo we need to care?all of us, yes. If there weren’t problems, we wouldn’t need these.attendees, yes. Know before you go.organizers, yes. Better to think about it ahead of time and have a plan than have to make up a strategy during an event if something happens.me, in particular, and Michael. Ugh. yes. our first meetup is next month. I’d like to be in line with the rest of Python. So, yep, we are going to have to talk about this and put something in place.Michael #3: The Interview Study Guide For Software EngineersA checklist on my last round of interviews that covers many of the popular topics.Warm Up With The ClassicsFizz Buzz560. Subarray Sum Equals KArrays: Left RotationStrings: Making AnagramsNth FibonacciMany many videos on interview topics and ideasData Structures Algorithms Big O NotationDynamic ProgrammingString ManipulationSystem DesignOperating SystemsThreadsObject OrientedDesign PatternsSQLFun conversation in the commentsBrian #4: re-assert : “show where your regex match assertion failed”Anthony Sotille“re-assert provides a helper class to make assertions of regexes simpler.”The Matches objects allows for useful pytest assertion messagesIn order to get my head around it, I looked at the test code:https://raw.githubusercontent.com/asottile/re-assert/master/tests/re_assert_test.pyand modified it to remove all of the with pytest.raises(AssertionError)… to actually get to see the errors and how to use it. def test_match_old(): > assert re.match('foo', 'fob') E AssertionError: assert None E + where None = [HTML_REMOVED]('foo', 'fob') E + where [HTML_REMOVED] = re.match test_re.py:8: AssertionError ____________ test_match_new ___________________ def test_match_new(): > assert Matches('foo') == 'fob' E AssertionError: assert Matches('foo') ^ == 'fob' E -Matches('foo') E - # regex failed to match at: E - # E - #> fob E - # ^ E +'fob'Michael #5: awesome-python-typingCollection of awesome Python types, stubs, plugins, and tools to work with them.TaxonomyStatic type checkersStub packagesToolsIntegrationsArticlesCommunitiesRelatedStatic type checkers: mypy - Optional static typing for Python 3 and 2 (PEP 484).Stub packages: Typeshed - Collection of library stubs for Python, with static types.Tools (super category): pytest-mypy - Mypy static type checker plugin for Pytest.Articles: Typechecking Django and DRF - Full tutorial about type-checking django.Brian #6: Developer Advocacy: Frequently Asked QuestionsDustin IngramI know a handful of people who have this job title. What is it? disclaimer: Dustin is a DA at Google. Other companies might be differentWhat is it? “I help represent the Python community at [company]"“part of my job is to be deeply involved in the Python community.”working on projects that help Python, PyPI, packaging, etc.speaking at conferencestalking to people. customers and non-customerstalking to product teamsbeing “user zero” for new products and featurespaying attention to places users might raise issues about productsworking in open sourcecreating content for Python devsbeing involved in the community as a company reprepresenting Python in the companycoordinating with other DAsWork/life?Not all DAs travel all the time. that was my main question.Talk Python episode: War Stories of the Developer EvangelistsExtras:https://www.meetup.com/Python-PDX-West/Michael:requests moves to PSFJoke:via https://twitter.com/NotGbo/status/1173667028965777410Web Dev Merit Badges

Sponsored by Datadog: pythonbytes.fm/datadogBrian #1: Dropbox: Our journey to type checking 4 million lines of PythonContinuing saga, but this is a cool write up.Benefits“Experience tells us that understanding code becomes the key to maintaining developer productivity. Without type annotations, basic reasoning such as figuring out the valid arguments to a function, or the possible return value types, becomes a hard problem. Here are typical questions that are often tricky to answer without type annotations:Can this function return None?What is this items argument supposed to be?What is the type of the id attribute: is it int, str, or perhaps some custom type?Does this argument need to be a list, or can I give a tuple or a set?”Type checker will find many subtle bugs.Refactoring is easier.Running type checking is faster than running large suites of unit tests, so feedback can be faster.Typing helps IDEs with better completion, static error checking, and more.Long story, but really cool learnings of how and why to tackle adding type hints to a large project with many developers.Conclusion. mypy is great now, because DropBox needed it to be.Michael #2: Setting Up a Flask Application in Visual Studio CodeVideo, but also as a postFollow on to the same in PyCharm:video and postSteps outside VS CodeClone repoCreate a virtual env (via venv)Install requirements (via requirements.txt)Setup flask app ENV variableflask deploy ← custom command for DBVS CodeOpen the folder where the repo and venv liveOpen any Python file to trigger the Python subsystemEnsure the correct VENV is selected (bottom left)Open the debugger tab, add config, pick Flask, choose your app.py fileDebug menu, start without debugging (or with)Adding tests via VS CodeOpen command pallet (CMD SHIFT P), Python: Discover Tests, select framework, select directory of tests, file pattern, new tests bottle on the right barBrian #3: Multiprocessing vs. Threading in Python: What Every Data Scientist Needs to KnowHow data scientists can go about choosing between the multiprocessing and threading and which factors should be kept in mind while doing so.Does not consider async, but still some great info.Overview of both concepts in general and some of the pitfalls of parallel computing.The specifics in Python, with the GILUse threads for waiting on IO or waiting on users.Use multiprocessing for CPU intensive work.The surprising bit for me was the benchmarksUsing something speeds up the code. That’s obvious.The difference between the two isn’t as great as I would have expected.A discussion of merits and benefits of both.And from the perspective of data science.A few more examples, with code, included.Michael #4: ORM - async ORMAnd https://github.com/encode/databasesThe orm package is an async ORM for Python, with support for Postgres, MySQL, and SQLite.SQLAlchemy core for query building.databases for cross-database async support.typesystem for data validation.Because ORM is built on SQLAlchemy core, you can use Alembic to provide database migrations.Need to be pretty async savyBrian #5: Getting Started with APIsdataquest.io postConceptual introduction of web APIsDiscussion of GET status codes, including a nice list with descriptions. examples:301: The server is redirecting you to a different endpoint. This can happen when a company switches domain names, or an endpoint name is changed.400: The server thinks you made a bad request. This can happen when you don’t send along the right data, among other things.endpointsendpoints that take query parametersJSON dataExamples in Python for using:requests to query endpoints.json to load and dump JSON data.Michael #6: Memory management in PythonThis article describes memory management in Python 3.6.Everything in Python is an object. Some objects can hold other objects, such as lists, tuples, dicts, classes, etc.such an approach requires a lot of small memory allocationsTo speed-up memory operations and reduce fragmentation Python uses a special manager on top of the general-purpose allocator, called PyMalloc.Layered managersRAMOS VMMC-mallocPyMemPython Object allocatorObject memoryThree levels of organizationTo reduce overhead for small objects (less than 512 bytes) Python sub-allocates big blocks of memory.Larger objects are routed to standard C allocator.three levels of abstraction — arena, pool, and block.Block is a chunk of memory of a certain size. Each block can keep only one Python object of a fixed size. The size of the block can vary from 8 to 512 bytes and must be a multiple of eightA collection of blocks of the same size is called a pool. Normally, the size of the pool is equal to the size of a memory page, i.e., 4Kb.The arena is a chunk of 256kB memory allocated on the heap, which provides memory for 64 pools.Python's small object manager rarely returns memory back to the Operating System.An arena gets fully released If and only if all the pools in it are empty.ExtrasBrian:Tuesday, Oct 6, Python PDX West, Thursday, Sept 26, I’ll be speaking at PDX Python, downtown.Both events, mostly, I’ll be working on new programming jokes unless I come up with something better. :)Michael:GitbookCall for Proposals for PyCon 2020 Is OpenJokes:A few I liked from the dad joke list.What do you call a 3.14 foot long snake? A π-thonWhat if it’s 3.14 inches, instead of feet? A μ-π-thonWhy doesn't Hollywood make more Big Data movies? NoSQL.Why didn't the div get invited to the dinner party? Because it had no class.

Sponsored by DigitalOcean: pythonbytes.fm/digitaloceanBrian #1: Annual Release Cycle for Python - PEP 602Under discussionAnnual release cadenceSeventeen months to develop a major version5 months unversioned, 7 months alpha releases when new features and fixes come in, 4 months of betas with no new features, 1 month final RCs.One year of full support, four more years of security fixes.Rationale/Goalssmaller releasesfeatures and fixes to users soonermore gradual upgrade pathpredictable calendar releases that line up will with sprints and PyConUSexplicit alpha phasedecrease pressure and rush to get features into beta 1RisksIncrease concurrent supported versions from 4 to 5.Test matrix increase for integrators and distributions.PEP includes rejected ideas like:9 month cadencekeep 18 month cadenceMichael #2: awesome-asgiA curated list of awesome ASGI servers, frameworks, apps, libraries, and other resources ASGI is a standard interface positioned as a spiritual successor to WSGI. It enables communication and interoperability across the whole Python async web stack: servers, applications, middleware, and individual components.Born in 2016 to power the Django Channels project, ASGI and its ecosystem have been expanding ever since, boosted by the arrival of projects such as Starlette and Uvicorn in 2018.Frameworks for building ASGI web applications.Bocadillo - Fast, scalable and real-time capable web APIs for everyone. Powered by Starlette. Supports HTTP (incl. SSE) and WebSockets.Channels - Asynchronous support for Django, and the original driving force behind the ASGI project. Supports HTTP and WebSockets with Django integration, and any protocol with ASGI-native code.FastAPI - A modern, high-performance web framework for building APIs with Python 3.6+ based on standard Python type hints. Powered by Starlette and Pydantic. Supports HTTP and WebSockets.Quart - A Python ASGI web microframework whose API is a superset of the Flask API. Supports HTTP (incl. SSE and HTTP/2 server push) and WebSockets.Responder - A familiar HTTP Service Framework for Python, powered by Starlette. (ASGI 2.0 only, ed.)Starlette - Starlette is a lightweight ASGI framework/toolkit, which is ideal for building high performance asyncio services. Supports HTTP and WebSockets.Brian #3: Jupyter meets the EarthLindsey Heagy & Fernando Pérez“We are thrilled to announce that the NSF is funding our EarthCube proposal “Jupyter meets the Earth: Enabling discovery in geoscience through interactive computing at scale” ““This project provides our team with $2 Million in funding over 3 years as a part of the NSF EarthCube program. It also represents the first time federal funding is being allocated for the development of core Jupyter infrastructure.”“Our project team includes members from the Jupyter and Pangeo communities, with representation across the geosciences including climate modeling, water resource applications, and geophysics. Three active research projects, one in each domain, will motivate developments in the Jupyter and Pangeo ecosystems. Each of these research applications demonstrates aspects of a research workflow which requires scalable, interactive computational tools.”“The adoption of open languages such as Python and the coalescence of communities of practice around open-source tools, is visible in nearly every domain of science. This is a fundamental shift in how science is conducted and shared.”Geoscience use casesclimate data analysishydrologic modelinggeophysical inversionsUser-Centered Developmentdata discoveryscientific discovery through interactive computingestablished tools and data visualizationusing and managing shared computational infrastructureMichael #4: Asynchronous Djangovia Jose NarioPython compatibilityDjango 3.0 supports Python 3.6, 3.7, and 3.8. We highly recommend and only officially support the latest release of each seriesThe Django 2.2.x series is the last to support Python 3.5.Other items butBig news: ASGI supportDjango 3.0 begins our journey to making Django fully async-capable by providing support for running as an ASGI application.This is in addition to our existing WSGI support. Django intends to support both for the foreseeable future.Note that as a side-effect of this change, Django is now aware of asynchronous event loops and will block you calling code marked as “async unsafe” - such as ORM operations - from an asynchronous context.Brian #5: The 1x EngineerFun take on 10x. List actually looks like probably a 3-4x to me, maybe even 8x or more. How high does this scale go anyway?non-exhaustive list qualities, here’s a few.Has a life outside engineering.Writes code that others can read.Doesn't act surprised when someone doesn’t know something.Asks for help when they need it.Is able to say "I don't know."Asks questions.Constructively participates in code reviews.Can collaborate with others.Supports code, even if they did not write it.Can feel like an impostor at times.Shares knowledge.Never stops learning. [obviously listens to Python Bytes, Talk Python, and Test & Code]Is willing to leave their comfort zone.Contributes to the community.Has productive and unproductive days.Doesn't take themselves too seriously.Fails from time to time.Has a favorite editor, browser, and operating system, but realizes others do too.Michael #6: Sunsetting Python 2January 1, 2020, will be the day that we sunset Python 2Why are you doing this? We need to sunset Python 2 so we can help Python users.How long is it till the sunset date? pythonclock.org will tell you.What will happen if I do not upgrade by January 1st, 2020? If people find catastrophic security problems in Python 2, or in software written in Python 2, then most volunteers will not help fix them.I wrote code in Python 2. How should I port it to Python 3? Please read the official "Porting Python 2 Code to Python 3" guide.I didn't hear anything about this till just now. Where did you announce it? We talked about it at software conferences, on the Python announcement mailing list, on official Python blogs, in textbooks and technical articles, on social media, and to companies that sell Python support.ExtrasBrian:working on a Portland Westside Python Meetup, info will be at pythonpdx.comHoping to get something ready for Oct. But… if not, hopefully by Nov.Michael:Humble Level Up Your Python Bundle

Sponsored by DigitalOcean: pythonbytes.fm/digitaloceanBrian #1: rapidtables“rapidtables … converts lists of dictionaries to pre-formatted tables. And it does the job as fast as possible.”Also can do color formatting if used in conjunction with termcolor.colored, but I’m mostly excited about really easily generating tabular data with print.Can also format to markdown or reStructured text, and can do alignment, … Michael #2: httpxA next generation HTTP client for Python. 🦋HTTPX builds on the well-established usability of requests, and gives you:A requests-compatible API.HTTP/2 and HTTP/1.1 support.Support for issuing HTTP requests in parallel. (Coming soon)Standard synchronous interface, but with [async](https://www.encode.io/httpx/async/)/[await](https://www.encode.io/httpx/async/) support if you need it.Ability to make requests directly to WSGI or ASGI applications.This is particularly useful for two main use-cases:Using httpx as a client, inside test cases.Mocking out external services, during tests or in dev/staging environments.Strict timeouts everywhere.Fully type annotated.100% test coverage.Lovely support for “parallel requests” without full asyncio (at the API level).Also pairs with async / await with async client.Plus all the requests featuresBrian #3: Quick and dirty mock service with StarletteMatt LaymanMock out / fake a third party service in a testing environment.Starlette looks fun, but the process can be used with other API producing server packages.We tell people to do things like this all the time, but there are few examples showing how to.This example also introduces a delay because the service used in production takes over a minute and part of the testing is to make sure the system under test handles that delay gracefully.Very cool, easy to follow write up. (Should probably have Matt on a Test & Code episode to talk about this strategy.)Michael #4: Mocking out AWS APIsvia Giuseppe CunsoloA library that allows you to easily mock out tests based on AWS infrastructure.Lovely use of a decorator to mock out S3Moto isn't just for Python code and it isn't just for S3. Look at the standalone server mode for more information about running Moto with other languages.Be sure to check out very important note.Brian #5: μMongo: sync/async ODM“μMongo is a Python MongoDB ODM. It inception comes from two needs: the lack of async ODM and the difficulty to do document (un)serialization with existing ODMs.”works with common mongo drivers such as PyMongo, TxMongo, motor_asyncio, and mongomock. (Hadn’t heard of mongomock before, I’ll have to try that out.)Note: We’ve discussed MongoEngine before. (I’m curious what Michael has to say about uMongo.)Michael #6: Single Responsibility Principle in Pythonvia Tyler MattesonI’m a big fan of the SOLID principlesThey even come in demotivator style postersDILiskov Substitution PrincipleOpen/Closed PrincipleSingle Responsibility PrincipleInterface Segregation PrincipleThis article will guide you through a complex process of writing simple code.ExtrasMichael:Code Challenge Bite 220. Analyzing @pythonbytes RSS feedJokes Q: What do you get when you cross a computer and a life guard?A: A screensaver!Q: What do you get when you cross a computer with an elephant?A: Lots of memory!via https://github.com/wesbos/dad-jokes Anti-joke (we ready for those yet?): A Python developer, a PHP developer, a C# developer, and a Go developer went to lunch together. They had a nice lunch and got along fine.

Sponsored by DigitalOcean: pythonbytes.fm/digitaloceanSpecial guest: Trey HunnerBrian #1: Positional-only arguments in Pythonby SanketFeature in 3.8“To specify arguments as positional-only, a / marker should be added after all those arguments in the function definition. “Great example of a pow(x, y, /, mod=None) function where the names x and y are meaningless and if given in the wrong order would be confusing. Trey #2: django-stubsType checking for Django!It’s new and from my very quick testing on my Django site it definitely doesn’t work with everything yet, but it’s promisingI don’t use type annotations in Django yet, but I very well might eventuallyMichael #3: CodeCombatSuper fun game for learning to codeReal code but incredibly easy codingCan subscribe or use the free tierJust got $6M VC fundingBrian #4: Four Use Cases for When to Use Celery in a Flask Applicationor really any web frameworkby Nick Janetakis“Celery helps you run code asynchronously or on a periodic schedule which are very common things you'd want to do in most web projects.”examples:sending emails outconnecting to 3rd party APIs. performing long running tasks. Like, really long. Running tasks on a schedule. Trey #5: pytest-stepsCreated by smarieCan use a generator syntax with yield statements to break a big test up into multiple named “steps” that’ll show up in your pytest outputIf one step fails, the rest of the steps will be skipped by defaultYou can also customize it to make optional steps, which aren’t required for future steps to run, or steps which depend on other optional steps explicitlyThe documentation shows a lot of different ways to use it, but the generator approach looks by far the most readable to me (also state is shared between steps with this approach whereas the others require some fancy state-capturing object which looks confusing to me)I haven’t tried this, but my use case would be my end-to-end/functional tests, which would work great with steps because I’m often using Selenium to navigate between a number of different pages and forms, one click at a time.Michael #6: docassembleCreated by Jonathan PyleA free, open-source expert system for guided interviews and document assembly, based on Python, YAML, and Markdown.FeaturesWYSIWYG: Compose your templates in .docx (with help of a Word Add-in) or .pdf files.Signatures: Gather touchscreen signatures and embed them in documents.Live chat: Assist users in real time with live chat, screen sharing, and remote screen control.AI: Use machine learning to process user input.SMS: Send text messages to your usersE-mail: Send and receive e-mails in your interviews.OCR: Use optical character recognition to process images uploaded by the user.Multilingual: Offer interviews in multiple languages.Multiuser: Develop applications that involve more than one user, such as mediation or counseling interviews.Extensible: Use the power of Python to extend the capabilities of your interviews.Open: Package your interviews and use GitHub and PyPI to share your work with the docassemble user community.Background Tasks: Do things behind the scenes of the interview, even when the user is not logged in.Scalable: Deploy your interviews on multiple machines to handle high traffic.Secure: Protect user information with server-side encryption, two-factor authentication, document redaction, and other security features.APIs: Integrate with third-party applications using the API, or send your interviews input and extract output using code.ExtrasMichael:PyPI closes in on 200kNumPy 1.17.0 releasedPython 3.8.0b4 is outJokevia Avram LubkinKnock! Knock!Who's there?Recursive function.Recursive function who?Knock! Knock!Nice. to get that joke, you’ll have to understand recursion. to understand recursion, either google “recursion”, and click on “did you mean “recursion””learn it in small steps. step one, recursiontext conversation:first person: “Hey, what’s your address?”second: gives IP addressfirst: “No. Your local address”second: 127.0.0.1first: “No. Your physical address”second: gives MAC address