Problem compiling Python 3.12 and up

Development
1

Hello,

I’m new here, and the post formatting is bewildering for me. Dunno if this the right forum to ask this. So sorry if there are errors.

So, anyone has tried to compile Python 3.12 and up, yet?

Requirements:

I have Searched the forum for my issue and found nothing related or helpful
I have checked the Resources category (Resources Index)
I have reviewed the Wiki for relevant information
I have read the the Release Notes and Errata

OpenMandriva Lx version:

OpenMandriva Lx release 25.06 (ROME) Rolling for x86_64

Desktop environment (KDE, LXQT…):

KDE (minimal)

Description of the issue (screenshots if relevant):

When trying to compile Python 3.12.x and 3.13.x with the source downloaded from the Official Python Site, it always fail at compiling/linking curses module.

Prior to compiling, I tried to install the dependencies with this command:

$ sudo dnf install make gcc patch automake findutils lib64bz2-devel lib64ffi-devel lib64gdbm-devel lib64lzma-devel lib64ncurses-devel lib64nsl-devel lib64openssl-devel lib64readline-devel lib64sqlite3-devel lib64tk-devel lib64uuid-devel lib64z-devel lib64zstd-devel

Relevant informations (hardware involved, software version, logs or output…):

This is the line where it fails (excerpt):

$ ./configure
$ make
...
gcc -D_DEFAULT_SOURCE  -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal -I./Include/internal/mimalloc  -I. -I./Include    -fPIC -c ./Modules/_cursesmodule.c -o Modules/_cursesmodule.o
gcc -shared      Modules/_cursesmodule.o -Os -fomit-frame-pointer -g3 -gdwarf-4 -Wstrict-aliasing=2 -pipe -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fstack-protector-all --param=ssp-buffer-size=4 -m64 -mtune=generic -flto -Oz -Wl,-O2 -Wl,--no-undefined -flto -lncursesw  -o Modules/_curses.cpython-313-x86_64-linux-gnu.so
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: Modules/_cursesmodule.o: in function `PyCursesWindow_set_encoding':
/home/xeon/Temp/Python-3.13.8/./Modules/_cursesmodule.c:2477:(.text+0x21): undefined reference to `PyUnicode_AsASCIIString'
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: /home/xeon/Temp/Python-3.13.8/./Modules/_cursesmodule.c:2480:(.text+0x32): undefined reference to `_PyMem_Strdup'
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: /home/xeon/Temp/Python-3.13.8/./Modules/_cursesmodule.c:2486:(.text+0x53): undefined reference to `PyMem_Free'
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: Modules/_cursesmodule.o: in function `Py_DECREF':
/home/xeon/Temp/Python-3.13.8/./Include/object.h:949:(.text+0x6c): undefined reference to `_Py_Dealloc'
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: Modules/_cursesmodule.o: in function `PyCursesWindow_set_encoding':
/home/xeon/Temp/Python-3.13.8/./Modules/_cursesmodule.c:2473:(.text+0x7b): undefined reference to `PyExc_TypeError'
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: /home/xeon/Temp/Python-3.13.8/./Modules/_cursesmodule.c:2473:(.text+0x8a): undefined reference to `PyErr_SetString'
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: /home/xeon/Temp/Python-3.13.8/./Modules/_cursesmodule.c:2483:(.text+0x96): undefined reference to `PyErr_NoMemory'
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: /home/xeon/Temp/Python-3.13.8/./Modules/_cursesmodule.c:2467:(.text+0x9f): undefined reference to `PyExc_TypeError'
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: /home/xeon/Temp/Python-3.13.8/./Modules/_cursesmodule.c:2467:(.text+0xae): undefined reference to `PyErr_SetString'
/usr/bin/x86_64-openmandriva-linux-gnu-ld.bfd: Modules/_cursesmodule.o: in function `PyCursesWindow_wtimeout':
...

I can post the full ./configure and make logs, but they are big (29.5Kb + 455.5Kb), so I ask first.

No matter the Python version and patch level (3.12x, 3.13x), it always choke at the same point.

Is it currently possible to compile Python v3.12 and up in OMLx, or is it not possible and the cause for the python version in the repo is v3.11.11?

Thanks

2

I guess there is a pretty good reason for not upgrading to latest available releases. I haven’t tried building python before. So cannot help with that. But if you are willing to experiment, there are two options.

  1. Take a look at OMLx python repo on GitHub. It seems to be updated to 3.13.3. There is a nice article on how to build packages for openmandriva. You may be able to build the package and install. Again remember, there must be a reason for not upgrading!
  2. Use homebrew (probably safer option). I recently discovered it while discussing on another thread. Once you have homebrew installed, install a particular version of python by appending @version. Like so, brew install python@3.13
    Afterwards, you can run the new version of python as python3.13 instead of just python or python3. It seems system installed version still remains default.
3

Well, the reason I’m compiling python mainly for use with PyEnv, or something similar. I don’t want to disturb the system installed python, and having a separate Python install other than from DNF/Repo install so I can manage it.

Also, I just migrate from Windows. I used Python 3.13.7 there.

In the mean time, I’m using the binaries from Python-Build-Standalone. But sadly, the Tcl/Tk included is v8.6 and compiled without Xft support, thus can only access a very small subset of the system installed font.

4

This isn’t a problem. I moved it to Development because we do already package python and you want to compile a newer version for yourself.

We are in the process of updating ROME and upgrading python is a topic of discussion for after that takes place. It will probably break a lot of things and we don’t have very many people to test it.

5

@n1ckla I was able to compile Python 3.14.

Here is what I did:

  • download the ‘python.spec’ file from the link I posted above. That is the OMLx official spec file. It has all the build dependencies.
  • Installed all the required dependencies via dnf builddep python.spec. You probably missed some dependencies.
  • Compiled the 3.14 version and install it.

I did get some error regarding pip installed to /usr/local/bin which is not in $PATH or something like that. But I was able to install 3.14 version and execute commands/scripts successfully.

6

Hi,

The problem is that you’re building the modules with checks for undefined symbols – _curses.cpython*.so fails because it references symbols defined in libpython without explicitly linking to it.

This is ok (because the module is never used outside of python), but the linker screams because it’s usually a very bad idea to to have unresolvable symbols in a library.

Possible fixes (any one of them should be sufficient):

  1. Link the module to the python library by adding -lpython3.13
  2. Tell the linker that it’s ok by adding -Wl,–allow-shlib-undefined to the compiler flags
  3. Use lld

The reason why we’re currently on 3.11 is simply that updating python to a new major version is just about the most painful thing there is (our package builds currently rely on mock, which is written in python and uses various extra python libraries that already need to be available for the version of python being used – so there’s a bit of a chicken and egg problem, we need to have those extra python modules before we can build a package, but we need to build the packages containing those python modules before we can get them into the build environment). We’re fixing that by getting rid of mock, there’s better ways to do what it does anyway.

We have a 3.14.0 package ready to go (spec file, adapted patches etc. are here), but we’re currently holding off on building it so we don’t delay the update sync any more. It will arrive in cooker shortly.

7

Thanks for the suggestion. I use your 3rd suggestion, and run configure like this:

$ ./configure --enable-optimizations --prefix=/home/xeon/Temp/py LDFLAGS=“-fuse-ld=lld”

The compilation/build passed, but one of the test suite fails:

...
test_win_redefine_function (test.test_sqlite3.test_userfunctions.WindowFunctionTests.test_win_redefine_function) ... ok
test_win_sum_int (test.test_sqlite3.test_userfunctions.WindowFunctionTests.test_win_sum_int) ... ok

======================================================================
FAIL: test_trace_too_much_expanded_sql (test.test_sqlite3.test_hooks.TraceCallbackTests.test_trace_too_much_expanded_sql)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xeon/Temp/Python-3.13.8/Lib/test/test_sqlite3/util.py", line 35, in wrapper
    func(self, *args, **kwargs)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xeon/Temp/Python-3.13.8/Lib/test/test_sqlite3/test_hooks.py", line 342, in test_trace_too_much_expanded_sql
    with self.check_stmt_trace(cx, expected):
         ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/xeon/Temp/Python-3.13.8/Lib/contextlib.py", line 148, in __exit__
    next(self.gen)
    ~~~~^^^^^^^^^^
  File "/home/xeon/Temp/Python-3.13.8/Lib/test/test_sqlite3/test_hooks.py", line 244, in check_stmt_trace
    self.assertEqual(traced, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: Lists differ: ["select 1 as a where a='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'/*+96 bytes*/"] != ['select 1 as a where a=?']

First differing element 0:
"select 1 as a where a='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'/*+96 bytes*/"
'select 1 as a where a=?'

- ["select 1 as a where a='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'/*+96 bytes*/"]
+ ['select 1 as a where a=?']

----------------------------------------------------------------------
Ran 494 tests in 0.959s

FAILED (failures=1, skipped=4)
test test_sqlite3 failed
0:01:11 load avg: 1.18 [36/43] test_sqlite3 failed (1 failure)
0:01:11 load avg: 1.18 [37/43] test_statistics
0:01:14 load avg: 1.16 [37/43] test_statistics passed
0:01:14 load avg: 1.16 [38/43] test_str
0:01:17 load avg: 1.16 [38/43] test_str passed
0:01:17 load avg: 1.16 [39/43] test_struct
0:01:18 load avg: 1.15 [39/43] test_struct passed
0:01:18 load avg: 1.15 [40/43] test_tabnanny
0:01:18 load avg: 1.15 [40/43] test_tabnanny passed
0:01:18 load avg: 1.15 [41/43] test_time
0:01:22 load avg: 1.15 [41/43] test_time passed
0:01:22 load avg: 1.15 [42/43] test_xml_etree
0:01:23 load avg: 1.14 [42/43] test_xml_etree passed
0:01:23 load avg: 1.14 [43/43] test_xml_etree_c
0:01:24 load avg: 1.14 [43/43] test_xml_etree_c passed

Total duration: 1 min 24 sec
Total tests: run=9,394 failures=1 skipped=208
Total test files: run=43/43 failed=1
Result: FAILURE
make: *** [Makefile:900: profile-run-stamp] Error 2

Well, it seems that the sqlite test fails. While I can live with that (but preferably if it passed), that also halts make install and make altinstall which apparently requires all the test suite to pass.

From the source directory, I can run the built python, and most the functionality I frequently use runs fine. Even IDLE and Tcl/Tk runs better than the binaries from Python-Build-Standalone.

Any way to fix this?

PS: My intention is to create a separate Python package that I can put in ./local/opt directory, then symlink the binaries to ./local/bin under alternate name (python to py3, pip to pylib3) as to not interfere with the one the system installed.

8

Check the sqlite project for patches.

9

Sorry for my ignorance, which sqlite project to check for patches: the official or OpenMandriva specific?

Because, prior to installing OpenMandriva on my main system, I’ve compiled the same Python version on MX Linux 21 (on my laptop) just a week prior, and it compiles flawlessly without any errors (no curses linking errors, no test errors, all tests including sqlite3 passes).

If all fails, does anybody has any ideas to bypass sqlite3 tests so it doesn’t halt the make install process?

10

OK, an update.

I tried to bypass the sqlite test by patching the Lib/test/test_sqlite3/test_hooks.py file:

Original:

@with_tracebacks(
    sqlite.DataError,
    regex="Expanded SQL string exceeds the maximum string length"
)
def test_trace_too_much_expanded_sql(self):

Patched:

@unittest.skip("SQLite 3.49.1 compatibility - trace format differs")
@with_tracebacks(
    sqlite.DataError,
    regex="Expanded SQL string exceeds the maximum string length"
)
def test_trace_too_much_expanded_sql(self):

make and make test now runs. However, using the --enable-optimizations means that the build system uses PGO optimization. It does this by:

  1. Compile python un-optimized
  2. Runs the test suite (43 cases) and profile
  3. Recompile everything using the profile in step 2
  4. Runs extensive test suite (480 cases)

With sqlite test case patched, step 2 runs without a hitch. However, I notice that in step 3, there are many warnings that the optimization process can’t find profile data for many modules. Also, step 4 takes a long time (especially when testing sockets), and yield to many errors:

Total duration: 13 min 20 sec
Total tests: run=46,930 failures=238 skipped=1,967
Total test files: run=484/480 failed=5 skipped=20 resource_denied=2 rerun=6
Result: FAILURE then FAILURE
make: *** [Makefile:2202: test] Error 2

The test failures are predominantly Tcl/Tk related, though there are others (python compile, etc).

Well, if this is what the maintainers faced, I symphatize with the pain. Apart from the “chicken and egg” and the “mock problem”, to ensure python compiles and works correctly must be a large pain in the behind.

For now, I’m disabling --enable-optimizations, and uses these steps to compile python:

$ ./configure CC=clang LDFLAGS="-fuse-ld=lld" --prefix=/home/xeon/Temp/py
$ sed -i 's/--no-undefined/--allow-shlib-undefined/g' Makefile
$ make
$ make install

There’s no test suite patching, as the test suite doesn’t run.

At least it build for me, and preliminary testing seems to run fine for my usage. YMMV.

Also, the built binaries are significantly larger than the downloaded version from Python Build Standalone Project: 239mb (clean compile, Clang 19.1.7) vs 124mb (downloaded binaries, IPython installed with pip, Clang 20.1.4). The culprit is that make install also copies the test suite (140mb), libpython3.13.a (37mb), and __pycache__(15mb). And then, the resulting python3.13binary is unstripped. Removing all that can yield to 130~140mb binaries (unstripped).

11

When we refer to the project specifically, we are always referring to the upstream project not our package. If you are compiling something from source we do not package (at all or at that version) It will be the best source to get information from.

MX Linux is based on Debian, and we are not. We don’t even package the same versions in Rock that they do in stable. Our versions will be newer. Therefore, this comparison (or any other distro for that matter) is invalid.

This is also something the upstream can answer faster than we can.

12

at_sigvirt, @bero, @zeroability:
Thanks for taking time to answer my questions. I’m able to compile Python 3.13.8. Your pointers helps immensely.

Though my compile may not be perfect, it’s good enough for now. I’ll report back if there are solutions for using --enable-optimization.