i am spending a little timecentsfocus right now converting some small language models to onnx

edit: note, the below models are pretty small. i’m curious to try onnx export with the 1M ctx 7b large world model. this may sadly be hard for me to return to due to the severity of personal difficulty i engaged. i’d like to try turning the system off mid-run (and losing work) more to get more comfortable with it when things come up.

first i tried converting a german finetune of gemma-2b (because the raw version required configuring an access token etc) using the basic hf optimum package

seems corrupt so i uploaded twice

{"id": "Drm97433RN_UoXFwkdC4KP1cih3zKQcLiMue5W6qVu8", "timestamp": 1710040253149, "version": "1.0.0", "public": "pGFsvdSB9sxbwU5L4HD2v12DK40kzZ5N69s6WlI3Uw9pFdHMHei3n1Tv4jvZqU9yeIMGsS60MQRvfJK1AEoNYsQqk4Rciajw0_IemZdwlt4u4voDALRalrQ3NV4knOlHRY11anqV0fNhikWCsiRPukIRZrdcFfqzFr0boH8bou7DgESNvWxROOxSC149oKxJ06FQsBDaIeElBsR8qTddybvXqMagXCM9y_HNrtAoz_8LgPjQtK5LFEbXhh9PyI_GOuoHyzJUc9Sm-V9kCB4kTm-SHrPbETQnvejZBcqEHxNcDNWBv6CWjj3-0V3dFMhjM1cy14d0Lm4j0IyRLm9bHM3s0ssVDd20gjWyar-D0o6guJIrteEC7UGR-w1yvXoGuIwdfZeoSAZ_CU9FrOJfQCTDs2aLgdCNeYKXg0Rt8YZL_elZnG7utCkO78TwxbGqear_I-1dlO39CUlo13YSS6pPonioWqkzXcXh93G7BYjgUxcPJ31kLyr2wBRA4OObAYRvh-5V3TkULlmwR4Q0pV3cUeOLI94b4WhaDZDI_RIJiCXQvtGy190NqTBeVogPrrAXLFkK0E013GByHrmzZoELfSUorjK-bDk4wXxdbVqzY7KXP-NEt3Bu-woinbUf56i3DXLrYlwINYK39VUydGpcQLZ5EDCL4u_IL_iFPt0", "signature": "hBEG3wURsaUDi3cnCgiXm-pgzCMaOu9u2jNrbhsbTy99HgXjMLt1onmdB7r2OANnrH2Vc3gzRvETkk0WWrAmdThXXj8GBMxV7F2l9mLYDe_gk49Vo91qNWPUOMiXECGIH5kLHNbxoaa4eYODjkASYJB3f3AmR6L5lrOyQAID2GZ33ZEhxpPXFvcUN1twFDtWTrBxtkRREAdm8YiiEn_eyvrf_mc0vwwSsCF87RoAG31UoJM54Mw_u1oQEuARMAx353vDtbDCUX-7B_uuHhY5VnfnWpRbvERnKjMveTGGE6JvuKJAiEgELKAMTEEnbEPkeWp6msW_o-wtkkO0AmzFxYaHnE-rhOPdtQeMTseeZEUxtRINZtbE7O9jXE8ZMXwqcts7Xzod175vut-XaNIvcT43ahEejsBEF-hDIR5nxYUF27WEDoame6XlbGhNLyYiyO8iOix8_ZdKKJp-vxVR2zF3p-Aej9rFCAkHrk_Rfi_E802wAnyQow3losx1ux5ouNo0i4ZSRXhaKhi_1v9EYmUkE47WQYlkbVHgcnORW4_iZ1dZP51Wv9PguRxhbsqTh58QA1msNf0KLJDuS7C3ZuNxcEDxQM4MsYR4T2xljA-Fa_QhGvbYhu34BIoz94pN405sf8EluZY9WJgYdd-XgDvxUies4DJtJsEIdLZ6jgw", "deadlineHeight": 1384377, "block": 1384377, "validatorSignatures": [], "name": "sauerkraut-2024-03-09.onnx.tar", "size": 12256921600, "start_height": 1380377, "start_block": "ndo_whntI2itZPNJcS0--8kD62TiWQD8ir1EMBaVOJlIJADtyZoNnaiKj_qzmbgC", "depth": 4} {"id": "zccCabIGaLTIpuVUdT3U2E2LE4GQ1BG7ruPtli3NFJ4", "timestamp": 1710052107096, "version": "1.0.0", "public": "pGFsvdSB9sxbwU5L4HD2v12DK40kzZ5N69s6WlI3Uw9pFdHMHei3n1Tv4jvZqU9yeIMGsS60MQRvfJK1AEoNYsQqk4Rciajw0_IemZdwlt4u4voDALRalrQ3NV4knOlHRY11anqV0fNhikWCsiRPukIRZrdcFfqzFr0boH8bou7DgESNvWxROOxSC149oKxJ06FQsBDaIeElBsR8qTddybvXqMagXCM9y_HNrtAoz_8LgPjQtK5LFEbXhh9PyI_GOuoHyzJUc9Sm-V9kCB4kTm-SHrPbETQnvejZBcqEHxNcDNWBv6CWjj3-0V3dFMhjM1cy14d0Lm4j0IyRLm9bHM3s0ssVDd20gjWyar-D0o6guJIrteEC7UGR-w1yvXoGuIwdfZeoSAZ_CU9FrOJfQCTDs2aLgdCNeYKXg0Rt8YZL_elZnG7utCkO78TwxbGqear_I-1dlO39CUlo13YSS6pPonioWqkzXcXh93G7BYjgUxcPJ31kLyr2wBRA4OObAYRvh-5V3TkULlmwR4Q0pV3cUeOLI94b4WhaDZDI_RIJiCXQvtGy190NqTBeVogPrrAXLFkK0E013GByHrmzZoELfSUorjK-bDk4wXxdbVqzY7KXP-NEt3Bu-woinbUf56i3DXLrYlwINYK39VUydGpcQLZ5EDCL4u_IL_iFPt0", "signature": "PoLCUzKdHJLzTYP1ig_dM14sGC4Be9qW_AzTVRNzfGWamE41Wi6t9mfSV0TurZQm6Ocm6pym_tVwypb9MHthjT5Zuj8DXnp0n9STGpUcLLKRBCO_udepSpNzSSk0N6VZIlDxZpXeDEcYU-nowvefYu5Y9N1gSLJzRY9N_F6KGE8u8T7F6H41q8qekC-9ffE63jWo9Ay4isDrA0z5RQ74M0DSwYjskRYLEpH1-U4O0ig1942sSgsHmucm_4Uo7204c9BY6S4f8XJzsKrwIxxKEeB1x0lOa5YdRk5xBDugP_MBtaqPPp7YT-DXS75SGE9RxJ1WyHf9lISHNFiQcj5ZYC1Rqt5H7j8FTA-flmsyMQbOTncbKeBR6Krrn-FpXja23Q5eblgq-UEL0rBUoTZF07hL72wPFIfWUJuxjpYIiiol5XwKe2aD9DeB-LO6tDA7aSLda2bu_eFK_LpnVVjoUT4QeYVTA9sNSUQvsjLyU8ou9yGdK7Nv8WS8bBKpEqUH3wJemGYdy9SiRE00DWecNCNJwOK39aS_bIJFEY1TO1UYsBwzlgAipidz3vJAljldMcVXmqVOLtOOYRFLkMlQBb9tuzq8jUjp3QwWhC-EtgrpEvgaHA5-0DJYPUr0CDGFoUW33jjZPpkvVogE5cwso3evhe0E49-TMr-N_eMf0vo", "deadlineHeight": 1384467, "block": 1384467, "validatorSignatures": [], "name": "sauerkraut-2024-03-09.onnx.tar.zst", "size": 4784945538, "start_height": 1380468, "start_block": "p2k_5kI0m8jblAlzaFDOQt6rXTwuC1qYAwTVee7nfb34SNxcWwhmuU-J6vjGIOIU", "depth": 4}

i’m looking for the microsoft olive formatted phi2 models :s i tried running the conversion in a machine with 32GB ram and it crashed

i tried running in a machine with >100GB ram but only 30GB disk space and it ran out of space and i didn’t see it using more than 32GB ram running.

i learned you might be able to pass —gpu to olive to speed up the compilation

following day. here is phi2 attempt with olive. may be corrupt. i’m having behavioral issues so stopping. https://github.com/microsoft/Olive examples/phi2

python3 phi2.py --model_type cpu_int4 --inference --max_length 16 --prompt once\ upon\ a

the cache folder was around 24GB but output model only 2.4GB

it took me like two days to send this email, during which my ipad wouldn”t power on and i could barely direct my body 0.0

{"id": "6YI2fCV0d1O_ZcMcOJqWmlLB_X1poQmdWIBbqbLzph4", "timestamp": 1710111615721, "version": "1.0.0", "public": "pGFsvdSB9sxbwU5L4HD2v12DK40kzZ5N69s6WlI3Uw9pFdHMHei3n1Tv4jvZqU9yeIMGsS60MQRvfJK1AEoNYsQqk4Rciajw0_IemZdwlt4u4voDALRalrQ3NV4knOlHRY11anqV0fNhikWCsiRPukIRZrdcFfqzFr0boH8bou7DgESNvWxROOxSC149oKxJ06FQsBDaIeElBsR8qTddybvXqMagXCM9y_HNrtAoz_8LgPjQtK5LFEbXhh9PyI_GOuoHyzJUc9Sm-V9kCB4kTm-SHrPbETQnvejZBcqEHxNcDNWBv6CWjj3-0V3dFMhjM1cy14d0Lm4j0IyRLm9bHM3s0ssVDd20gjWyar-D0o6guJIrteEC7UGR-w1yvXoGuIwdfZeoSAZ_CU9FrOJfQCTDs2aLgdCNeYKXg0Rt8YZL_elZnG7utCkO78TwxbGqear_I-1dlO39CUlo13YSS6pPonioWqkzXcXh93G7BYjgUxcPJ31kLyr2wBRA4OObAYRvh-5V3TkULlmwR4Q0pV3cUeOLI94b4WhaDZDI_RIJiCXQvtGy190NqTBeVogPrrAXLFkK0E013GByHrmzZoELfSUorjK-bDk4wXxdbVqzY7KXP-NEt3Bu-woinbUf56i3DXLrYlwINYK39VUydGpcQLZ5EDCL4u_IL_iFPt0", "signature": "R37ajj8lQ5_lbsLJ69kDlLg-r1sAc75QTidHNO6y3LOgp96gIbSLY7vBlv2fLlzDX25KJdIQ3Tg3Bv0Pdqnx1l1fstXKRGMN-fm0wf5vVcrHR0mI5vKppZpVfJWEwO1iRioO3nllJ2Iv6rm38gRvXe8D2eERL0kFBJ1fx5wZR3GT8lie4Ccq7_g1PRpjSsuVv-y-3JUlG0Bkpu7-8fTp1iOOp-k1oog3wvUGwhIPlwgY_ySZ_OsXuL-wcclsDJf8GMByPaCU3XXlQD26RMXtD6w-crliYDkrMhin3t9j_VzZCxVoSWgtI4WJk6ut080v48pSBwCaylNQbtX0d_qT-BUOEUzStCQlUiv0X1ZW4T5OYU1Ajd5mHfLrtODnJKcaLDSF3WzNFEBzZjlOVuJIXXEL-dseK5uipTperP4cpu-b8HqbUpAiVgVKYarT93be0hwZiqAPxdR9X-FZOBrw6YFgEUv0N8Ps7BXkHzpjTA7K5Z0QG2TFCE6hPUpJpeM2Ty68jJJzD48XOtSoAb-06OICF3TAPFc3w2oUqQEto1-9aVuJ6m76DwybbVG2zPOasHvjO9UlGojNJuUkjhoktzoR6CtEIgOgc9N6SUabIC7iLkYbh-ECQJa1lWIFVcajmlzZB8Y1QZ26ZjotMMPcBV0Ox8-kXBxkCm-MTcrKQ-o", "deadlineHeight": 1384926, "block": 1384926, "validatorSignatures": [], "name": "phi2_cpu_int4_onnx.tar", "size": 2525132800, "start_height": 1380926, "start_block": "efoRyvFvQ38DZjyRZCIVanR8XDguu0WOlUP2bhJImTupG44Zui--L7qK0UI1nvj5", "depth": 4}