{ "version": "https://jsonfeed.org/version/1.1", "title": "Blog", "home_page_url": "https://blog.schuetze.link", "feed_url": "https://blog.schuetze.link/feed.json", "language": "en", "authors": [ { "name": "Konstantin Schütze" } ], "items": [ { "id": "https://blog.schuetze.link/python-ecosystem-statistics/", "url": "https://blog.schuetze.link/python-ecosystem-statistics/", "title": "Python Ecosystem Statistics", "content_html": "
\n\n
\n
\n

Python Ecosystem Tool and Build Backend Statistics

\n

Last refreshed: 2026-04-16

\n\n
\n

\n Aggregating data on the usage of tools and build backends over time,\n using the grep.app search, the source distributions of the top PyPI\n packages, and the PyPI Linehaul BigQuery data.\n

\n
\n
\n
\n\n
\n

Datasets

\n
\n

\n Linehaul: Each PyPI simple API access and\n file download is logged to\n BigQuery through\n linehaul. The metadata includes package name/version, installer info,\n subcommand, and CI status. The data is reliably available since 2019,\n with CI tracking being added in 2024 (via CI=1). Poetry\n doesn't report CI info. We use 7-day and 90-day rolling averages to\n smooth daily fluctuations.\n

\n

\n Build backends: Analysis of the\n top 15k PyPI packages\n by downloads. Build backend info is extracted from\n pyproject.toml and setup.py in source\n distributions. The setup.py category tracks packages\n without a [build-system] table. 1 of 15,000 packages\n failed to analyze and was excluded.\n

\n

\n grep.app: Uses search hits for different\n tools on grep.app, which indexes public\n code repositories. While smaller than GitHub, it accurately reports\n hit counts. According to a\n Vercel blog post, grep.app includes \"1M+ pre-indexed repos\". Search hits are captured\n daily.\n

\n
\n
\n\n
\n

Minimum Python Version

\n\n
\n \n \n \n
\n \n Figure 1: Minimum Python version (bars) from requires-python in the\n latest version of the top 15,000 PyPI packages by downloads in the\n last 30 days. The orange line shows the cumulative distribution\n among packages, which is the share of packages can be installed with\n the given Python patch version.\n \n
\n
\n
\n\n
\n

Build System Distribution

\n\n
\n \n \n \n
\n \n Figure 2: Build backend distribution among the top 15k PyPI packages\n by downloads, comparing all packages vs. those with uploads in the\n last 365 days.\n \n
\n
\n
\n\n
\n

Code Search Statistics

\n\n
\n \n \n \n
\n \n Figure 3: Search hits for the main commands for pip (top) and\n non-pip package managers (bottom) in grep.app, recorded daily.\n \n
\n
\n\n
\n \n \n \n
\n \n Figure 4: Growth in the search hits for package managers in\n grep.app, recorded daily, relative to start of the recording. The\n dotted lines are idealized projections from linear fits.\n \n
\n
\n
\n\n
\n

PyPI Downloads

\n\n
\n \n \n \n \n \n \n
\n \n Figure 5: Daily wheel and source distribution downloads by tool as\n tracked by linehaul, in absolute numbers (top) and relative\n fractions by tool (bottom). Only tools with >0.1% usage and only\n tools reporting linehaul data are shown.\n \n
\n
\n
\n\n
\n

Astral Static Tools

\n\n
\n \n \n \n
\n \n Figure 6: Daily search hits for the main ruff and ty subcommands in\n grep.app. Linear fits for each tool are shown as dotted lines.\n \n
\n
\n
\n\n
\n

uv subcommands

\n\n
\n \n \n \n
\n \n Figure 7: Wheel and source distribution downloads by uv subcommand\n in the last 7 days, split by CI vs. non-CI usage. No subcommand\n generally means an old version that doesn't report the subcommand.\n \n
\n
\n\n
\n \n \n \n
\n \n Figure 8: Simple API requests by uv subcommand in the last seven\n days, split by CI vs. non-CI usage. No subcommand generally means an\n old version that doesn't report the subcommand.\n \n
\n
\n\n
\n \n \n \n
\n \n Figure 9: Wheel and source distribution downloads by uv version last\n 7 days, split by CI vs. non-CI usage. Only the 20 most used versions\n are shown.\n \n
\n
\n\n
\n \n \n \n
\n \n Figure 10: Cumulative distribution of downloads by uv version age,\n split by CI vs. non-CI usage, using downloads in the last 7 days.\n This is approximately the time since the last version update,\n discounting the time since the latest release.\n \n
\n
\n\n
\n \n \n \n
\n \n Figure 11: Cumulative distribution of downloads by version age,\n comparing uv, pip, and poetry, using downloads in the last 7 days.\n This is approximately the time since the last version update,\n discounting the time since the latest release. For poetry, data is\n only reported since linehaul support was added.\n \n
\n
\n
\n\n
\n

Figures not shown

\n
\n

Two figures were excluded from this page for lack of signal.

\n
    \n
  • \n CI vs. non-CI downloads from PyPI don't show a clear trend in the\n short timespan they cover, ranging between 20% and 35% CI downloads\n (Figure).\n
    • \n
  • \n The downloads of the most popular PyPI segmented by tool have a high\n variance with no discernible pattern (Figure).\n
    • \n
\n
\n
\n\n
\n

(Intermediate) Data

\n
\n

The intermediate data aggregated data used to generate plots.

\n \n
\n
\n
\n", "date_published": "2026-04-16T00:00:00+00:00", "tags": [ "python" ] }, { "id": "https://blog.schuetze.link/reimplementing-pep-440/", "url": "https://blog.schuetze.link/reimplementing-pep-440/", "title": "Reimplementing PEP 440", "content_html": "

I've reimplemented PEP 440, the python version standard, for monotrail: pep440-rs. Did you now that 1a1.dev3.post1+deadbeef is a valid python version, there's not only == but also === and that version specifiers are context sensitive?

\n

Let's start with the normal stuff: There are basic version numbers with dots in between (like 2.3.1) and optionally alpha/beta/release candidate suffixes (canonically 2.3.1b1, but conveniently lenient so 2.3.1-beta.1 also works). For dependencies, there operators for minimums and maximums separated by comma such as >=2.5.1,<3. You can of course also select a specific prerelease (e.g. 1.1a1 being matched by ==1.1a1) and maybe you've also seen constraints like 1.2.*. But below the clear semver-y surface lie many demons of the old.

\n

\"A

\n

Photography by Norbert Buduczki

\n

It all starts with the part of the version that's hidden in the default: The epoch. By default it's zero, but if you want to switch versioning system, you can add the new epoch with an exclamation mark like 1!4.2.0. Since version ordering is defined as a total order, 2020.1 < 1!0.1.0, but also for some reason <1!0.1.0 matches 2020.1 and >2020.1 match 1!0.1.0 (the specifiers are not a total order normally, I don't know why it doesn't specify to never match across epochs). Being a mere mortal, I have never witnessed the turning of epoch myself, but the feature remains part of The Old Code.

\n

You can also add .dev and .post with some number to all versions, e.g. 1.0.0.dev1 or 1.0.0.post1. Or just combine them and do 1.0.0.post1.dev1, which is a developmental release of a post-release. That of course doesn't stop at final releases, you can now do 1.0.0a1.post1.dev1 to have a developmental release of a post release of a prerelease (in canonical form alpha/beta/rc don't have a dot, but dev and post do, while also in PEP 440 dev release are sometime included with the prereleases). If you sort them, obviously 1.0.0.dev1 < 1.0.0 < 1.0.0.post1, and 1.0.0a1.dev1 < 1.0.0a1 < 1.0.0a1.post1 < 1.0.0. But dev releases of the final version are sorted lower than any prerelease version, so suddenly we have 1.0.0.dev1< 1.0.0a1.dev1 < 1.0.0a1 < 1.0.0, while also having 1.0b2 < 1.0b2.post345.dev456. That is, the try-out release for 1.0 proper is considered older than the try-out release for the 1.0 alpha. The only sensible way to implement this sorting is making a five-tuple where you map (pre-releasity, pre-number, post-number or None as smallest, dev-number or int max as largest, local version) and let tuple-sorting sort out the rest. Even pypa/packaging uses tuple logic feat. ±infinity.

\n

Matching version with specifiers such as >=1.2.0 or <2.0.0 is tricky because PEP 440 says \"Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release\". That's really fuzzy, and also it means whether a single version matches a specifier depends on the environment, something I confused myself with. pypa effectively says that you must add to specifier whether you want to match prereleases (which here again include dev releases) or not when using the library. The consequence is that when you say ~=2.2 but there's only a 2.2.1a1 it will pick that alpha version (but not 2.2a1, which never matches).

\n

There are also local versions which can added with a + after the regular version, such as 3.4.0+my.local.123.version. The 123 is going to get ordered as a number, everything that can't be parsed as a number will get ordered as a string. Information about usage is sparse, apparently linux distributions use it to tag their python packaging. Those are also in semver, but more reasonable as \"build metadata\": \"Build metadata MUST be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence\".

\n

Finally, there's also ===, \"Arbitrary equality\", which is advertised as \"simple string equality operations\" that \"do not take into account any of the semantic information\". pypa/packaging has a test that ===lolwat parses with the comment \"=== is an escape hatch in PEP 440\".

\n
\n

For those wondering why python1 didn't pick a sane standard like semver to begin with, the basic syntax format for writing things like >1.0, !=1.3.4, <2.0 was written down in PEP 314 in 2003 (!) 2. PEP 386, the first python version standard, was written in 2009, \"codifying existing practices\" and its successor and current standard PEP 440 in 2013. In comparison, the first commit to npm was made in 2010, semver v1.0.0 was published in 2011, v2.0.0 in 2013, npm inc. was founded in 2014 and cargo had its first commit also in 2014. So python has a hard time doing modern packaging because they were trying to do modern packaging before it was being invented.

\n

I still believe that (a) bringing in features from semver and tools such as poetry, cargo and npm would greatly benefit the python ecosystem and (b) python packaging isn't doomed to stay in its current state. While e.g. pypi's backend will have to handle everything that ever used to be legal, i believe that the ecosystem at large can and must migrate to better tools and standards. This is largely informed by having to deal with a lot of the breakages of the current state of python packaging and trying to support friends and colleagues.

\n

The easiest is probably to deprecate ===, even PEP 440 soft-deprecates it with \"Use of this operator is heavily discouraged and tooling MAY display a warning when it is used\".

\n

For epochs, i haven't seen them used even a single time. To try to make this at least a bit empirical i ran two queries on the pypi bigquery data3 4, with the result that in one month there were 19,699,031,713 downloads, 40,281 of which for versions specifying an epoch, that's 0.0002%.

\n

Post releases can be replaced with publishing a new patch release or a one-higher pre-release. Historically, it was a good idea that you could specify 1.2.3 and if the author messed up 1.2.3 and has to publish fixup wheels you'd be directly moved to the fixup, but nowadays you want lock files where this doesn't work anymore, and it also interacts weirdly with yanking. This applies especially to post releases of prereleases, which PEP 440 acknowledges: \"Creating post-releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create a new pre-release by incrementing the numeric component\".

\n

Dev version on prereleases seem also strange to me (just publish a higher prerelease instead, a test-release of a test-release is kinda redundant). For dev versions of final releases there are certainly workflows that benefit from them5, even though other ecosystems do fine without special casing .dev. The main problem are the strange semantics, and while PEP 440 defends this as \"far more logical sort order\", i strongly disagree, this was and is super confusing, and also the implementation is a mess. When removing dev (and ideally also post) releases at least for alpha/beta/rc versions, the semantics would become intuitive again, with dev release simply being a prerelease one level below alpha releases6.

\n

For local version the semver style \"purely informative and no semantics\" definition would be imho more reasonable; i unfortunately can't tell if de-semanticizing local version would break anything (as in, is anybody currently depending on the fact that 1.0+foo.10 has precedence over 1.0+foo.9).

\n

Given that pip now has a backtracking dependency resolver, i think we can simplify the spec a lot by separating it into three parts: One part that defines the version number schema and precedence (a total order as it currently is), one part that translates operators such as ~= into normal >/=/> sets that directly translate to the version order, and one part that specifies the rules for resolvers, that is when are they allowed to pick which prerelease. The latter isn't well-defined as of PEP 440, but imho we should agree about this across the ecosystem7. See e.g. node on prereleases and cargo on prereleases8. I particularly like the node/npm \"If a version has a prerelease tag (for example, 1.2.3-alpha.3) then it will only be allowed to satisfy comparator sets if at least one comparator with the same [major, minor, patch] tuple also has a prerelease tag\". For comparison, firefox estimates 16–20 minutes for the semver spec, but 57–73 minutes for PEP 440.

\n

For all change there would need to be long announcement and deprecation periods with a specific focus on helping people migrate their workflows. For the deprecation period, tools should print big red warnings whenever they encounter something broken. Speaking of announcements, there's really a lack of an official pypa communication channel! An official blog for announcements on deprecations, changes, release, and (proposed) PEP status changes together with a community aggregator like This Week in Rust would be extremely helpful over the current word-of-mouth-in-twitter-replies-and-buried-github-issues system.

\n

Two features that would be great to add are the caret operator (^) and the tilde operator (~) from semver. Nowadays semver is arguably the most popular version scheme even in python, and for most packages you want ^1.2.3 and for the remainder (including calver projects that treat the last digit as semver-like patch version) ~1.8 will do the right thing. I'd like to add them to pep440-rs eventually but i'm neither sure about the exact semantic yet nor how to let users switch between PEP 440-only specifiers and the modern superset.

\n

Next Up: PEP 508

\n
\n
    \n
  1. \n

    Well, technically not python as the python interpreter but pypa as the vague group of people who make the packaging PEPs. Python itself didn't even have a concept of package versions at all until importlib.metadata introduced optionally reading a version as a string to the standard library, and the language itself still doesn't have a concept of packages but merely one of modules. When you import foo it effectively just asks sys.meta_path if anyone can import foo, which will check if any location in sys.path has a foo module, but this has no relation to packaging. If you ask stdlib's importlib.metadata for an installed package version, it really just asks sys.meta_path with a different method if anyone optionally wants to tell it about the package version, which by default will just look for .dist-info folders in your sys.path. ↩

    \n
    2. \n
  2. \n

    If you ever wondered why wheel metadata is in some archaic e-mail-headers RFC 822

    \n
    STANDARD FOR THE FORMAT OF\nARPA INTERNET TEXT MESSAGES
    \n

    that's because it was picked in 2001. Even XML 1.0 was published just 3 years prior. I'm still very much in favor of migrating to a JSON or TOML format such as pkg-info.json or editing pyproject.toml similar to what cargo does, but that's for another time.

    \n ↩
    3. \n
  3. \n

    I ran this on 2022-11-29 and the queries were

    \n
    SELECT COUNT(*)\nFROM bigquery-public-data.pypi.file_downloads\nWHERE timestamp BETWEEN\n  TIMESTAMP(DATETIME_SUB(CURRENT_DATETIME(), INTERVAL 1 MONTH))\n  AND TIMESTAMP(CURRENT_DATETIME())
    \n

    and

    \n
    SELECT COUNT(*)\nFROM bigquery-public-data.pypi.file_downloads\nWHERE timestamp BETWEEN\n  TIMESTAMP(DATETIME_SUB(CURRENT_DATETIME(), INTERVAL 1 MONTH))\n  AND TIMESTAMP(CURRENT_DATETIME())\n  AND CONTAINS_SUBSTR(file.version, '!')
    ↩
    4. \n
  4. \n

    Blessed be whoever came up with the bigquery datasets for pypi ↩

    \n
    5. \n
  5. \n

    E.g. some people want to build {Major}.{Minor}.{Patch}.dev{YYYY}{MM}{DD}{MonotonicallyIncreasingDailyBuildNumber} in their CI workflows. Local versions are used to indicate when linux distributions did some downstream packing, so you can directly tell when you're looking at a distro patched install. ↩

    \n
    6. \n
  6. \n

    I'm still not sure if they provide any benefit over just using alpha versions, but once they behave like normal prereleases their implementation and cognitive overhead is near zero so backwards compatibility is way more significant. Note that semver relies on \"alpha\", \"beta\" and \"rc\" being alphabetically ordered, while we need to make \"dev\" lowest manually, otoh semver also allows any random stuff for prereleases and uses the same duck-typed logic for comparing them as PEP 440 uses for local versions. ↩

    \n
    7. \n
  7. \n

    Consider the case where a user adds a library A from pypi that has multiple transitive dependencies on B, some specifier with preleases in their specifiers and some without. It would be bad for the authors of A to work if they couldn't clearly reason which prereleases of B might or might not be picked independent of which tool the user uses. ↩

    \n
    8. \n
  8. \n

    According to the python survey results, those are the two most popular other package managers in use\n\"Plot\nRubyGems states \"The RubyGems team urges gem developers to follow the Semantic Versioning standard for their gem's versions. The RubyGems library itself does not enforce a strict versioning policy, but using an \"irrational\" policy will only be a disservice to those in the community who use your gems\", but i couldn't find any details on what versions and operators are allowed.\nComposer on the other hand is very much like python (docs): \"This must follow the format of X.Y.Z or vX.Y.Z with an optional suffix of -dev, -patch (-p), -alpha (-a), -beta (-b) or -RC.\", where dev is below alpha. It also seems to allow 1.2.* but i couldn't find any more documentation on what's allowed and what the semantics are except that they apparently transform prereleases to a version digit ↩

    \n
    9. \n
\n
\n", "date_published": "2022-12-01T00:00:00+00:00", "tags": [ "python", "rust" ] }, { "id": "https://blog.schuetze.link/a-dive-into-packaging-native-python-extensions/", "url": "https://blog.schuetze.link/a-dive-into-packaging-native-python-extensions/", "title": "A dive into packaging native python extensions", "content_html": "

The complete guide to building you own native wheel from scratch

\n

There are cases where you want to extend python with native code, e.g. for scientific computing (numpy, scipy), database connectors (mysqlclient, psycopg2) or UI (pygobject, pyqt). For cpython this is traditionally done in C/C++, but you can also use the C api from D (pyd), go (cffi) or rust (cffi or pyo3).

\n

Distributing those extensions is a big problem. Until recently, the only viable option was to write special plugins for setuptools, e.g. milksnake for cffi or setuptools-rust for pyo3. Inspired by the new pyproject.toml, I wanted to get rid of the flaws and resulting pain of setuptools. So I went to write pyo3-pack, which aims at making packaging and publishing native python modules in rust as easy as wasm-pack makes it for javascript.

\n

It turns out that writing such a tool is relatively easy (less than a thousand lines of rust to get from source to wheel). The hard part is to find out what you need to do in the first place. The documentation is scattered across different and partially outdated tutorials, PEPs, stack overflow answers, references, examples and source code; I sometimes even had to resort to reverse engineering. So I decided to write down everything I learned about native wheels, which eventually became this blog post.

\n

The good parts

\n

The official tutorial on native modules, Extending Python with C or C++, is a good introduction to the core concepts of native modules: The header files, the calling conventions, GC, the object protocol and error handling. It only shows building for C/C++ with distutils (the predecessor to setuptools) though, omits the officially blessed manylinux, and lacks an explanation of the abi and linking options (more on those later).

\n

For the daily work, the Python/C API Reference Manual is often much better. It also has some explanation for the ABI.

\n

For the rest of the post I'll assume that you have built your native python module as shared library (e.g PyInit_<modname> function for python 3) with your technology of choice.

\n

Metadata

\n

Each python package, whether it is an egg or a wheel or a source archive, is described by structured metadata, which contains fields required for pip to work and informational fields used e.g. for pypi.

\n

There are five versions for the metadata of python packages: 1.0 (PEP 241), 1.1 (PEP 314), 1.2 (PEP 345), 2.0 (PEP 426) and 2.1 (PEP 566).

\n

2.0 was an attempt to replace the key-value structure of the metadata with a json like structure. This could have been a big improvement, but was withdrawn (and is not accepted by pypi or pip) since it would have been a to big breakage. This is why the current version is called 2.1, even though it is backwards compatible to 1.0.

\n

The current specification can be found at PyPA's Core metadata specifications page, which is pretty self-explaining and worth reading.

\n

N.B.: https://www.pypa.io/en/latest/roadmap/ is completely outdated as it still features as Metadata 2.0 as part of the roadmap. https://packaging.python.org/specifications/core-metadata/#description is misleading since you must not use the RFC 822 in the metadata for the pypi upload (see the section on uploading) and for the METADATA file inside the wheel you can just put the description in the body, i.e. after all the keys.

\n

Tags and naming

\n

Native modules need to specify with which platforms and python interpreters they are compatible. Python has two coexisting standards with slightly different syntax: PEP 425 for packages and PEP 3149 for shared libraries. Both are based on abi tags, so let's discuss them first.

\n

The cpython ABI

\n

cpython abi is composed of the major and minor version of cpython and a set of abiflags, which are determined by compiler flags. According PEP 3149 there are three such compile time options we need to consider (at least for linux and mac):

\n\n

For practical purposes, d is irrelevant, m is always set and u may or may not be set - more on u below. The tag for this abi is cp{major}{minor}{abiflags} or cpython-{major}{minor}{abiflags}. My python 3.6 installation is for example cp36m and cpython-36m.

\n

The u or wide-unicode flag is about the representation of unicode characters (introductory article). Initially, python unicode characters were fixed to two bytes (UCS-2), meaning that any 3 or 4 byte characters were not representable. This changed with PEP 261, which added optional support for wide unicode characters (UCS-4) to python2. The choice between UCS-2 and UCS-4 was made a compile time option, creating the abi without \"u\" for UCS-2 and one with \"u\" for UCS-4 (ignoring the option to completly disable unicode). In python 3.3 this was replaced by a system that determines the representation at runtime described in PEP 393, removing the \"u\" flag from the abi. This means that the wide-unicode option is only relevant for backwards compatibility with python 2.

\n

The stable abi

\n

There are obviously some big drawbacks from having tons of different abis which you all need to support and build and test, so PEP 384 introduced the \"stable abi\" in python 3.2. This abi with the tag abi3 contains a subset of the full abi and that is forward compatible with all future 3.x releases of cpython. In the header files, everything that is not part of the stable abi is gated with #if !defined(Py_LIMITED_API).

\n

The stable abi is extended from time to time, meaning that you can require the stable abi and a minimum version. In the header files this is done by setting Py_LIMITED_API to the minimum support python in the PY_VERSION_HEX format as described in the documentation. In the headers this is checked e.g. with #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03030000 for a function that was added to the stable abi in python 3.3.

\n

Sysconfig

\n

In the initial version of this post, I wrote about getting the required information about the interpreter through sysconfig. But it turned out that sysconfig behaves inconsistently across python versions and operating systems. E.g. the VERSION field on linux is in the format {major}.{minor}, while it is {major}{minor} on windows (both with python 3.7). There's also EXT_SUFFIX, which tells you the complete extension of the library filename on linux (e.g. \".cpython-35m-x86_64-linux-gnu.so\"), but on windows it's just .pyd. I've collected a few samples in a folder in the pyo3-pack repo. You'll find more of those weird cases in there.

\n

I'm currently using the following snippet with python -c and do the logic and sanity checks in rust.

\n
import sysconfig\nimport sys\nimport json\n\nprint(json.dumps({\n    "major": sys.version_info.major,\n    "minor": sys.version_info.minor,\n    "abiflags": sysconfig.get_config_var("ABIFLAGS"),\n    "m": sysconfig.get_config_var("WITH_PYMALLOC") == 1,\n    "u": sysconfig.get_config_var("Py_UNICODE_SIZE") == 4,\n    "d": sysconfig.get_config_var("Py_DEBUG") == 1,\n    # This one isn't technically necessary, but still very useful for sanity checks\n    "platform": sys.platform,\n}))
\n

This is than deserialized into the equivalent of the following python 3.7 code:

\n
@dataclass\nclass Interpreter:\n    major: int\n    minor: int\n    abiflags: Optional[str]
\n

If you still want to use sysconfig, the easiest way is through python -m sysconfig. As seen above, you can use WITH_PYMALLOC (1 means m), Py_UNICODE_SIZE (4 means u) and Py_DEBUG (1 would mean d) for the python 2 abiflags.

\n

To get the flags in machine readable as Dict[str, Union[str, int]], use:

\n
python -c "import json, sysconfig; print(json.dumps(sysconfig.get_config_vars()))"
\n

For a Dict[str, str], use:

\n
python -c "import json, sysconfig; print(json.dumps({k:str(v) for k, v in sysconfig.get_config_vars().items()}))"

Naming shared libraries

\n

PEP 3149 defines that shared libraries will get a tag between the file name and the extension, separated by dot. It tells you that this tag needs to include at least the implementation (i.e. cpython) with its major and minor version. It also shows .cpython-32mu.so as an example for such file extension, from which we can derive .cpython-{major}{minor}{abiflags}.so as template.

\n

This sounds nice, but is extremely misleading if not plainly wrong in reality.

\n

From picking apart other native libraries and trial and error with filenames I figured the following:

\n\n

The only place the new, 3.5+ schema has ever been announced were the python 3.5 release notes. But rejoice, even those are wrong. (I tried googling both the wrong and the correct version, but it really seems to be only in those release notes)

\n

For 3.5+, I found that the following is what's actually working (and also what setuptools produce):

\n

Linux

\n

Template: .cpython-{major}{minor}{abiflags}-{architecture}-{os}.so

\n

architecture is either i386 or x86_64, and os is linux-gnu. The release notes state that the file extension is .pyd, which is wrong and doesn't work in practice. Also note that os has an internal minus, breaking the general rule of separating parts of the tag with a minus.

\n

Example: steinlaus.cpython-35m-x86_64-linux-gnu.so

\n

Mac OS

\n

Template: .cpython-{major}{minor}{abiflags}-darwin.so

\n

Example: steinlaus.cpython-35m-darwin.so

\n

Windows

\n

Template: {name}.cp{major}{minor}-{platform}.pyd

\n

The platform is either win_amd64 or win32. .pyd files are just renamed .dll files, which is confirmed in the official windows FAQ (which is otherwise extremely outdated)

\n

Example: steinlaus.cp35-win_amd64.pyd

\n

Naming wheels

\n

The documentation for defining wheels is much better than the one for naming so files with most parts being specified in PEP 425.

\n

The official schema from that PEP is {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl, which is used for all python versions. The distribution is you package's name escaped with re.sub(\"[^\\w\\d.]+\", \"_\", distribution, re.UNICODE), we can ignore and skip the build tag, the python tag for our case is cp{major}{minor}{abiflags}, the abi tag is either the python tag, abi3 or none.

\n

For the platform tag it states that \"The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _.\" This is unfortunate, since the output of distutils.util.get_platform() isn't specified, so we need to reverse engineer. Looking only at 32-bit and 64-bit x86, we have either win_amd64 or win32 for windows. For linux, we have linux_i686 or linux_x86_64, even though in practice we must use either manylinux1_i686 or manylinux1_x86_64 as desribed in the manylinux paragraph below. For mac the tag used by setuptools is macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64 for whatever reason.

\n

Examles:

\n
steinlaus-1.0.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
steinlaus-1.0.0-cp36-cp36m-manylinux1_x86_64.whl
steinlaus-1.0.0-cp36-cp36m-win_amd64.whl

Manylinux

\n

Libraries and binaries on Linux are traditionally (for the better or worse) dynamically linked to libraries in $LD_LIBRARY_PATH, which are installed through the systems package manager. Native modules could require arbitrary versions of arbitrary libraries, but can't guarantee they are installed on the target machine, leading to linker errors when importing. To avoid such incompatibilities, PEP 513 specifies a target manylinux1 which contains only a set of old versions of libraries that can be found on basically every Linux. (This is an extremely short summary of the rational in the PEP)

\n

Wheels for the manylinux1 target must be in the manylinux1 docker container. This container is based on CentOS 5, i.e. some very old Linux. Using this docker image is the only officially blessed way to build for the linux target in general. Pypi only accept wheels with the manylinux1 tag and rejects those with a linux tag. A slightly more modern target, manylinux2010 is currently being working on as a successor for manylinux1 (PEP 571 - The manylinux2010 Platform Tag, tracking issue).

\n

manylinux is accompanied by a tool called auditwheel that checks the library and then \"awards\" the manylinux1 tag. Afaik this is not checked by pypi, so it's possible to lie about that check.

\n

By default rust only links very few system libraries, which are a subset of the manylinux1 target. This means that pyo3-pack only needs to check that the constraints are met and we can otherwise totally skip the whole ancient-docker-mess.

\n

The internals of a (binary) wheel

\n

While there alternative ways to install python packages, using wheels with pip is (for good reasons) the officially blessed one, so for pyo3-pack I've only looked into into building wheels. They are specified in PEP 427.

\n

Wheels are generally just zip files with a .whl extension. They come in two flavors: sdist and bdist. bdist (\"built distribution\") wheels are pre-built packages. Their installation is mostly just unpacking the archive. They specify the compatible python version(s), an abi and a platform. sdist (\"source distribution\") wheels contain all the sources including your setup.py or pyproject.toml, so for installing them, they need to be built first.

\n

Every wheel contains a {distribution}-{version}.dist-info folder with the following files inside it, where {distribution} is again the name with the underscore-escapes.

\n\n

For actual package you have two options:

\n

If you only need to package one shared library, you put it at the top level of the zip. The shared library must be named according to the rules describe above, while basename must be the name of the module.

\n

Example:

\n
.\n├── get_fourtytwo-1.6.8.dist-info\n│   ├── METADATA\n│   ├── RECORD\n│   └── WHEEL\n└── get_fourtytwo.cpython-36m-x86_64-linux-gnu.so
\n

For any wheels containing python files, whether they have native components or not, the top level module is a python module. This means a directory at the top level with the name of the module and a __init__.py inside that direct ry. Inside this directory the same rules as for any other python project apply. Native modules work the same way as pure python single file module, only that the filenames end with .so or .pyd instead of .py. Take a look at numpy's wheels for a complex, real world scenario.

\n

Example:

\n
.\n├── get_fourtytwo\n│   ├── __init__.py\n│   ├── native_fourtytwo.cpython-36m-x86_64-linux-gnu.so\n│   └── python_fourtytwo.py\n└── get_fourtytwo-1.6.8.dist-info\n    ├── METADATA\n    ├── RECORD\n    └── WHEEL
\n

... where __init__.py contains

\n
from .native_fourtytwo import native_class\nfrom .python_fourtytwo import some_class
\n

Besides the presented wheel 1.0 format, PEP 491 defining a \"Wheel 1.9\" format also exists. It is officially in draft status, but it seems completely abandoned, with no mention neither on the mailing list nor in the relevant github repos. The PEP doesn't explain why version 1.9 should follow version 1.0.

\n

Note that you can lie to pypi about the metadata. E.g. I actually ran into a case, where a .tar.gz was uploaded as 3.0.{date}, while the installed package identified itself as 3.0.dev0, which didn't exist on pypi. This effectively broke pip freeze.

\n

Source distributions

\n

Source distribution, sdist for short, are special source archives that can be build and installed with pip. They are used e.g. when there are no wheels for the current platform/abi and as base for building debian or fedora packages. While they existed for a longer time, they are formally specified in PEP 517. This PEP differentiates between a source tree, which would be the git repository, and a source distribution, which this paragraph is about.

\n

A source distribution is a .tar.gz archive. It is explicitly stated that zip archives are not allowed anymore, even though it mentions lxml-3.4.4.zip as an example in the beginning. The filename is {name}-{version}.tar.gz. The archive contains one folder, which is named {name}-{version}. This folder contains the required source, a setup.py and/or a pyproject.toml, and a file called PKG-INFO which identical to the METADATA file in wheels.

\n

Example:

\n
foobar-0.11.2/\n├── foobar\n│   ├── __init__.py\n│   └── main.py\n├── LICENSE\n├── PKG-INFO\n├── pyproject.toml\n└── setup.py
\n

If the archive contains a pyproject.toml with a [build-system] section that specifies a list of packages required for building as requires and the path to a build backend object in build-backend, this backend should be called by pip to build the source distribution into a wheel. pip 10.0.1, which is the latest version as of this writing, refuses to install such wheels stating \"This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.\". We can therefore skip any further details about the build backend because we can't use it yet anyway.

\n

Without a pyproject.toml with those entries, pip executes the setup.py in the directory, meaning that currently the way to support source distributions is to use setuptools, which is exactely what I wanted to avoid. This means no sdist in custom packagers for now.

\n

As a side note, both flit and poetry already implement the PEP 517 interface (buildapy.py in flit and api.py in poetry) and add a pyproject.toml to the archive. But as they omit the [build-system], pip instead uses the setup.py they also create.

\n

Finding python interpreters

\n

It's convenient for building and essential for testing to find the installed python versions. For linux and mac, you can check which python binaries are in PATH and then use the snippet from above to get the version and abiflags. (Or you can use a fixed list of 2.7 and 3.5+ and just try each because there's no good library to work with PATH yet).

\n

For windows, every python version is just called python.exe. Fortunately, there is a launcher called py. With -0 (but not --list, even if the help says otherwise) it will list all known versions, which you can launch with py -{version}. It's then easy to get the path of the actual interpreter with py -{version} -c \"import sys; print(sys.executable)\".

\n

Contemporary legacy uploading

\n

Now that we've got our wheel built, we also want to publish it, i.e. upload it to pypi (which is now powered by a software called warehouse). It turns out that the api to upload packages is called the \"legacy api\", even though there's no new api for uploads (there is a json api, but it only supports reading package metadata). The upload part of the legacy api had no documentation other than \"use twine\", so I read through the source of poetry uploader, warehouse's endpoint and warehouse's tests to figure out how to use that api. Eventually I wrote a pull request to warehouse documenting that api.

\n

Errors

\n

As mentioned in the preface, all the information presented here is assembled from many different sources of varying qualitity and up-to-dateness, with some parts being reverse-engineered. So if you find any errors or missing parts, please ping me (konstin@mailbox.org, konstin on github, @konstinx on twitter).

\n", "date_published": "2018-07-21T00:00:00+00:00", "tags": [ "python", "rust" ] }, { "id": "https://blog.schuetze.link/meine-stadt-transparent-teil-2/", "url": "https://blog.schuetze.link/meine-stadt-transparent-teil-2/", "title": "Meine Stadt Transparent, Teil 2: Die Technik", "content_html": "

Seit fast vier Monaten entwickeln Tobias Hößl und ich Meine Stadt Transparent (Demo, GitHub). Im ersten Teil ging es darum, wie es dazu kam und warum wir dieses Projekt machen. In diesen Teil geht es um den momentanen Status und die technische Details.

\n

Ziele

\n

Wie in Teil 1 beschrieben gibt es bereits genügend kommerzielle Systeme, die sich hauptsächlich an Verwaltungsmitarbeiter und Stadträte richten, aber für Bürger kaum benutzbar sind. Wir wollen die Lücke eines bürgerfreundlichen Systems füllen. Anders gesagt wollen wir Zugänglichkeit für alle schaffen.

\n

Verständliche Sprache: Existierende Systeme sind oft eine Mischung aus unverständlichen Abkürzungen und Wortmonstern wie „Beschlussvollzugkontrolle\", „Ratsinformationssystem\" und „Anliegenmanagement\", die die meisten Nutzer erst mal googlen müssen. Das macht Thema „Stadtpolitik\" nur noch unsexier als es sowieso schon ist. Wir vermeiden deswegen Abkürzung, Fachbegriffe und Techspeak soweit möglich.

\n

Googlebarkeit: Informationen, die man durch die Suche nicht findet, existieren nicht. Natürlich kann man auch versteckte Daten finden, aber das ist Zeitaufwendig und scheitert oft auch einfach daran, dass man nicht weiß, was überhaupt existiert.

\n

Likability: Politik ist eines dieser Themen, die zwar sehr wichtig aber auch sehr langweilig sind. Deshalb ist es umso wichtiger, dass man eine Website dazu auch benutzen möchte. Technische Perfektion ist schön und gut, aber wenn eine Webseite so interessant ist wie ein Ladebalken wird sich kaum einer freiwillig dort aufhalten. Das heißt nicht, dass wir Seite mit Achievements, Kalendarsprüche und Animationen vollstopfen, aber ein paar Katzenbilder und Eastereggs dürfen es schon sein.

\n

Barrierefreiheit: Politik betrifft alle Menschen. Leider werden die Dokumente in der Regel nur als pdf veröffentlicht, woran wir aber im Moment leider nichts ändern können. Wir versuchen deshalb zumindest die Seite an sich durch html-aria-tags und Linter möglichst Barrierefrei zu halten.

\n

Nachhaltigkeit: Wenn Software nicht so gebaut ist, dass sich gut weiterentwickeln und damit an neue technische Entwicklungen anpassen lässt, veraltet sie und verliert dadurch nach und nach ihre Nutzbarkeit.

\n

\"Likability

\n

Likability durch cat content (Symbolbild)

\n

Frameworks und Bibliotheken

\n

Um diese Ziele umzusetzen brauchen wir einen ganzen Stapel an Technik.

\n

Das Backend ist in pyhton 3 mit django geschrieben. Als Datenbank verwenden wir Mariadb bzw. teilweise sqlite und für die Suche elasticsearch. Synchronisiert werden die Daten über django-elasticsearch-dsl, als Server läuft gunicorn hinter einem nignx. Für das Fontend nutzen wir Bootstrap 4 in django-Templates, dazu etwas javascript in einer npm-es6-babel-webpack-pipeline.

\n

Wir hatten am Anfang überlegt, ein Javascript-Framework zu verwenden, uns dann aber dagegen entschieden. Mit Angular hatten wir beide bereits Erfahrung, wollten es aber aber aus unterschiedlichen Gründen nicht verwenden, genauso wenig verwenden wie vue.js noch React. Bei den meisten anderen Frameworks muss man davon ausgehen, dass sie in der schnelllebigen Javascript-Welt untergehen und damit nicht für ein langfristiges Projekt taugen, wie der Stack Overflow Blog sehr anschaulich gezeigt hat. (Immer noch aktuelle Leseempfehlung dazu: How it feels to learn JavaScript in 2016). Ich persönlich würde am liebsten elm oder rust/webassembly verwenden, leider sind beide aber nicht annähernd ausgereift genug.

\n

Nach 3 Monaten mit Bootstrap bin ich mit unserer Entscheidung immernoch glücklich; Die Dokumentation von Bootstrap ist exzellent, die fertigen Klassen nehmen einem eine Menge Arbeit ab und für zwei nicht-Designer sieht die Seite nicht schlecht aus. Das Design kann später über Themes gut anpasst werden, was z.B. praktisch ist, um die Seite an verschiedene Städte anzupassen. Für Javascript-Bibliotheken brauchen wir außerdem keine Framework-Anbindung.

\n

Die Live Demo

\n

Wir haben eine Live-Demo unter meine-stadt-transparent.de, die echte Daten der Stadt Jülich anzeigt.

\n

Dank des großartigen Github Auto Deploy ist die Demo immer auf dem aktuellen Stand. Das heißt aber auch, dass ein falscher Commit die Seite zerschießen kann (und momentan auch darf).

\n

Der Datenimport

\n

Die Seite ist so gebaut, dass man mit etwas eigenem Python-Code prinzipiell beliebige Daten importieren kann. Damit das Projekt aber nicht nur einen hypothetischen Zweck sondern auch eine realen praktische Nutzen hat, haben wir einen Importer für OParl-Schnittstellen geschrieben. OParl spezifiziert dabei eine API, mit der Daten aus deutschen Ratsinformationssystemen (RIS) in einem einheitlichen json-Format über eine REST-Schnittstelle exportiert werden können. OParl 1.0 wurde zwar komplett ehrenamtlich entwickelt, wird aber mittlerweile von vier großen RIS-Herstellern angeboten bzw. entwickelt. Durch Open.NRW gibt es mittlerweile eine offizielle OParl-Unterstützung in 21 Kommunen mit Sternberg SD.NET. Eine der 21 Kommunen, die Kleinstadt Jülich, nutzen wir für unsere Demo-Seite.

\n

Den Importer zu schreiben war komplexer als gedacht und wir hatten kompliziertere Bugs und andere Probleme. Eines der gößeren Probleme war und ist die Integration von liboparl, da die Integration von gnome-Bibliotheken wie liboparl in Python deutlich schlechter funktioniert als erwartet. (Bedeutet in der Praxis, dass wir python-gobject/gi verwenden müssen, das aber nur mit System-Python und einem symbolischen Link in die virtualenv funktioniert. Das undokumentierte Über-pip-Installieren geht zwar theoretisch, in der Praxis dann aber doch nicht)

\n

Die Installation

\n

Meine Stadt Transparent soll sich möglichst einfach aufsetzen lassen. Dafür haben wir eine Schnellstart-Anleitung im readme, die mittels docker compose alle benötigten Dienste (im Moment mariadb, elasticsearch und django) einrichtet, startet und verbindet. Wegen schlechten Managments bei Docker und dem Datenimport ist immernoch etwas Handarbeit dabei, die wir aber so gering wie möglich halten.

\n

Ausblick

\n

Wir haben noch etwa zwei Monate bis zum Demoday am 28. Februar. Die Hauptaufgaben bis dahin sind ein OParl-Export, eine Änderungshistorie für alle wichtigen Objekte sowie ein paar interne Umbauten.

\n", "date_published": "2017-12-20T12:00:00+00:00", "tags": [ "ris" ] }, { "id": "https://blog.schuetze.link/meine-stadt-transparent-teil-1/", "url": "https://blog.schuetze.link/meine-stadt-transparent-teil-1/", "title": "Meine Stadt Transparent, Teil 1: Was bisher geschah", "content_html": "

Seit fast vier Monaten entwickeln Tobias Hößl und ich Meine Stadt Transparent (Demo, GitHub). In diesem ersten Blogpost geht es darum, wie es dazu kam und warum wir dieses Projekt machen. Im zweiten Teil wird es dann um das eigentliche Projekt und dessen technische Details gehen.

\n

\"Aktenstapel,

\n

Stadträte, Beezirksausschüsse und Gemeindräte produzieren seit langem große Mengen Papier. Der Münchner Stadtrat und die 25 Bezirksauschüsse allein haben bespielweise im Jahr 2015 wurden über 15.000 Dokumente mit mehr als 64.000 Seiten veröffentlicht.1 Um der großen Anzahl an Dokumenten Herr zu werden hat man in den 2000ern einen großen Teil der Stadtratsverwaltung digitalisiert. Dazu wurden spezielle Webportale gebaut, die Ratsinformationssysteme genannt werden. Während Städte besonders in der Anfangszeit auf Eigenentwicklungen oder Spezialanfertigungen gesetzt haben, haben sich mittlerweile fertige Systeme und eine handvoll Anbieter etabliert. Durch solche fertigen Systeme setzen mittlerweile auch immer mehr kleine Kommunen oder Kommunalverbände ein Ratsinformationssystem ein.

\n

Deren öffentliche Oberfläche ist oft die einzige Möglichkeit, an wichtige Dokumente zu kommen. Die Angelenheiten des eigenen Stadt- oder Gemeinderats betreffen eine mehr, als man oft erwartet. Dort wird z.B. jegliche Bauvorhaben, Schulen und Kitas, den ÖPNV sowie Unterstützung für eine Vielzahl von Vereinen und Initiativen entschieden. Leider sind die Webseiten als Informationsquellen für Bürger nicht zu gebrauchen: Die Oberflächen sind völlig veraltet, voll mit bürokratischen Fachbegriffen und außerdem voller technischer Fehler. Das größte Problem ist jedoch meisten das Fehlen einer Suche, mit der man etwas findet. Durch diese Probleme werden viele wichtige öffentliche Dokumente für die Öffentlichkeit faktisch unzugänglich.

\n

Das Problem haben Leute in verschiedenen Städten erkannt und versucht, auf eigenen Faust und fast immer ohne Unterstützung der Verwaltungen bessere Webseiten zu bauen. Diese Arbeit fand zu großen Teilen unter dem Dach der Open Knowledge Labs statt. Leider sind die meisten dieser Projekte nie fertig geworden und irgendwann eingeschlafen. Zwei Projekte waren jedoch erfolgreich und existieren bis heute.2

\n

Das eine ist Politk bei Uns, welches ursprünglich 2012 als Plattform für das Ruhrgebiet veröffentlicht wurde.3 Mittlerweile kann man unter anderem die Dokumente für Köln, Bochum und die Berliner Bezirke durchsuchen. Das Hauptziel ist es, möglichst viele Städte durchsuchbar und damit für Bürger (insbesondere auch über google und co.) zugänglich zu machen. Die offiziellen Daten werden automatisiert aufbereitet, indem z.B. der Text aus pdfs extrahiert wird und Adressen erkannt werden. (Von Politik bei Uns wird mittlerweile Version 2 entwickelt. Aber das ist eine eigene Geschichte.)

\n

Das andere Projekt ist München Transparent. Angefangen hat es als eine bessere Oberfläche für das münchner Ratsinformationssystem, die wie Politik bei Uns die Daten aufbereitet und durchsuchbar macht. Dort werden aber nicht nur nicht alle Daten des offiziellen Systems abgebildet, sondern auch zusätzliche Informationen wie Verordnungen, verständliche Erklärungen und Geodaten. Der Clou des Systems sind jedoch die Benachrichtigungen, die Nutzer per E-Mail informieren, wenn es in seiner Umgebung oder zu von ihm abonnierten Themen Neuigkeiten gibt.

\n

Der größte Teil wurde Tobias Hößl entwickelt, erst unter dem Namen \"OpenRIS\", dann als \"Ratsinformant\". Ende 2014 bin ich zu dem Projekt gestoßen und habe die Seite erweitert und verbessert. Im Februar 2015 haben wir die Seite dann als München Transparent richtig veröffentlicht. Mittlerweile benutzen sogar Angestellte der Stadt und einige Stadträte München Transparent statt des hauseigenen Systems.4 Mit der Stadt arbeiten wir auch seit längerem gut zusammen, wofür wir sehr dankbar sind.5 Verteilte Zuständigkeiten und langwierige Prozesse machen aber natürlich in der Stadtverwaltung vieles komplizierter machen als bei einem Hobbyprojekt.

\n\n\n\n\n\n\n\n\n\n
\"Die\"Die
\"Die\"Die
\n

Links ist München Transparent, recht das offizielle münchner Ratsinformationssystem, jeweils mit Links zu den ensprechenden Seiten.

\n

Beide Projekte nutzen sogenannte Scraper, um die Informationen der offiziellen Seiten auszulesen. Das sind mehr oder weniger kleine Programme, die jede Seite der Oberfläche abrufen, die interessanten Daten extrahieren und die Ergebnisse in einer Datenbank speichern. Aus dieser Datenbank wird dann wiederum die neue Webseite erstellt.

\n

Durch Scraping kommt man zwar an die benötigten Daten, aber eigentlich ist es ziemlich unsinnige Sache: Man steckt viel Arbeit in ein Programm, das im Endeffekt die Datenbank hinter einer Webseite rekonstruiert, in dem es jede Seite dieser Webseite aufruft. Viel einfacher wäre es, wenn man die Daten direkt in maschinenlesbarer Form bekommen könnte, am besten noch für jede Stadt im gleichen Format. Aus diesem Grund ist OParl entstanden. OParl ist eine maschinenlesbare Schnittstelle, die man mit geringem Aufwand in existierende Ratsinformationssysteme einbauen kann. Dazu werden die Daten verschiedener Städte auf eine gemeinsame Darstellung gebracht und es ist festgelegt, wie man diese Daten effizient kopieren kann.

\n

Über München Transparent wurden wir irgendwann auf OParl aufmerksam, dass damals noch eine ziemliche Baustelle mit immer wieder verschobenen Veröffentlichungsterminen war. Da ich eine solche Schnittstelle für München Transparent haben wollte (und weil ich damals noch zu viel Zeit und hatte), fing ich an mih bei OParl zu beteiligen. Im Juli dieses Jahres haben wir die erste Stabile Version 1.0 veröffentlicht, die im Moment von mehreren großen Anbietern in ihre Ratsinformationssysteme eingebaut wird. Politik bei Uns, München Transparent und 21 Kommunen in Nordrhein-Westfalen haben die Schnittstelle bereits. Eine Version 1.1 mit Fehlerkorrekturen und eine englische Übersetzung sind in Arbeit.

\n

Bei München Transparent haben wir verschiedene Anfragen bekommen, ob wir sowas wie München Transparent denn nicht auch für andere Städte machen könnten, schließlich ist die Oberfläche schon fertig und man bräuchte nur noch die Daten einer anderen Stadt dahinter zu setzen. Leider sind aber viele Teile des Programmcodes und unseres Datenmodells stark auf München angepasst. Dazu kommen andere technische Probleme, so ist das Framework, die technische Grundlage der Webseite, veraltet, und es fehlt an Tests zur Qualitätssicherung und der Code hat sehr viele Altlasten. Unserer Oberfläche schadet das glücklicherweise nicht, aber wir können sie dadurch nicht einfach auf andere Städte übertragen. Außerdem verhindern diese Altlasten langfristig eine vernünftige Weiterentwicklung.

\n

Deshalb hatte ich schon länger die Idee, das ganze mit den bisherigen Erfahrungen komplett neu zu schreiben. Mit einem normalen Alltag hat man nur leider nicht die Zeit dafür, und so blieb das lange nichts als eine schöne Idee. Das hat sich jedoch durch den Prototype Fund geändert. Der Prototype Fund ist eine Initiative der Open Knowledge Foundation Deutschland, bei der sich Projekte um eine Förderung von (damals) bis zu 30.000€ Euro über 6 Monate bewerben können. Das Geld dazu kommt vom Bundesministerium für Bildung und Forschung. Bei der zweiten Runde im März 2017 habe ich mich mit der Idee beworben, ein neues nutzerfreundliches Ratsinformationssystem zu entwickeln, Arbeitstitel \"Open Source Ratsinformationssystem\". Dieses soll sich im Gegensatz zu München Transparent in ganz Deutschland einsetzen lässt und auf einem soliden technischen Unterbau stehen. Daraus ist mittlerweile das Projekt \"Meine Stadt Transparent\" geworden, an dem ich zusammen mit Tobias Hößl seit fast vier Monaten arbeite. Darüber, wie es zur Zeit bei dem Projekt steht und was noch geplant ist, werde ich im zweiten Teil schreiben.

\n
\n
    \n
  1. \n

    Die Zahlen stammen aus der Datenbank von München Transparent ↩

    \n
    2. \n
  2. \n

    Es gibt natürlich auch noch andere Projekte, die erfolgreich ihr Ratsinformationssystem bereichern, wie z.B. Dresden Ratsinfo. Ich meine in diesem Fall aber nur solche, die eine weitestgehend vollständige, bürgernutzbare Oberfläche bauen. ↩

    \n
    3. \n
  3. \n

    OpenRuhr – offene Daten für das Ruhrgebiet ↩

    \n
    4. \n
  4. \n

    In Anträgen und Anfragen wird oft auf andere Dokumente bezug genommen. Dabei wird dann auch gerne München Transparent verlinkt, wie eine Meta-Suche auf München Transparent zeigt. ↩

    \n
    5. \n
  5. \n

    Dank eines Stadtratsantrags hat diese Zusammenarbeit auch ein politisches Mandat. ↩

    \n
    6. \n
\n
\n", "date_published": "2017-12-20T00:00:00+00:00", "tags": [ "ris" ] } ] }