Spaces:
Sleeping
Sleeping
Commit History
-
iq2_xxs: tune quantization (llama/5320)
Kawrakow +1: ikawrakow Unverified signatureWe get slightly better PPL, and we cut quantization time in nearly half. The trick is to 1st quantize without forcing points onto the E8-lattice. We can then use a narrower search range around the block scale that we got that way. Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>11e5f6b
-
cuda : fix LLAMA_CUDA_F16 (llama/5262)
slarenUnverified signature5fd8fb7
-
metal : add im2col F32 dst support (llama/5132)
Unverified signature26aec77 -
llava : add MobileVLM support (llama/5132)
JidongZhang-THU +1: slarenUnverified signature* New Feature: 1. Sum_Rows: fix cuda kernel overflow fix block shape error when nrows too big 2. Im2Col: Support Batch in cuda Support f32 to f32 both in cpu && cuda 3. DepthWiseConv: Support by Im2Col && MulMat 4. Pool_2d: Supoort avg pooling in cuda 5. HardSigmoid: Imp in cuda 6. HardSwish: Imp in cuda * fix tabs instead of spaces * code clean * CUDA POOL2D * ADD POOL2D test case in test-backend-ops.cpp * code clean * fix pool2d_kernel nits * fix bug in pool2d kernel * fix avg pooling, count_include_pad nits * test-backend-ops : add more pool_2d tests * cuda : fix warnings and formatting * ggml : check types in release builds too in pool_2d * test-backend-ops : remove f16 pool_2d tests * cuda : more style fixes * Add assert in ggml_cuda_op_pool2d * pool2d float padding fallback * test-backend-ops : add dst_type to im2col --------- Co-authored-by: slaren <slarengh@gmail.com>f17a416 -
ggml : limit n_threads to the max n_tasks (llama/5238)
slarenUnverified signature2645c33 -
kompute : llama-bench support and ggml_cpu_has_kompute() (llama/5226)
Unverified signature0c9c434
-
ggml : add abort_callback for cpu backend (ggml/725)
Michael PodvitskiyUnverified signature* a way to use abort_callback with the cpu backend * whisper updatea8ea91b
-
extra : update sync scripts
Unverified signatured99e873
-
server : allow CORS request with authorization headers (#1850)
Valentin GosuUnverified signatureWhisper plugin in Obsidian requires an API key which is then sent as an authorization header. However, the presence of an authorization header requires a CORS Preflight, so both the OPTIONS method and the Access-Control-Allow-Headers: authorization must be handled.16a6639 -
whisper.android : how to build with CLBlast (#1809)
Unverified signature* FetchContent * OpenCL * Documentation and make optional * Specify GGML build options in build.gradle * Use gradle properties * @ggerganov Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * @gpokat --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>eea7f53 -
whisper : expose CUDA device setting in public API (#1840)
Didzis GoskoUnverified signature* Makefile : allow to override CUDA_ARCH_FLAG * whisper : allow to select GPU (CUDA) device from public APId13ee66 -
make : add macOS deployment target option (#1839)
Didzis GoskoUnverified signature9c90601
-
talk-llama : stream response (#1121)
Unverified signature2193f2b
-
sync : ggml (#0)
Unverified signaturefded75b -
ggml : fix IQ3_XXS on Metal (llama/5219)
Kawrakow +1: ikawrakow Unverified signatureCo-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>f066321 -
sync : ggml (llama/0)
Unverified signaturecdb7964 -
Faster AVX2 dot product for IQ2_XS (llama/5187)
Kawrakow +2: ikawrakow, PeterReid Unverified signature* iq2xs: faster AVX2 dot product * iq2xs: small AVX2 imrovement * Speed up computing sign bits in AVX2 iq2_xs dot product --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com> Co-authored-by: Peter Reid <peter@peterreid.net>187ae44 -
SOTA 3-bit quants (llama/5196)
Kawrakow +1: ikawrakow Unverified signature* iq3_xxs: quantize/dequantize RMSE seems a bit high-ish at about half-way between q2_K and q3_K, so need to check more. * iq3_xxs: CUDA dequantize works * iq2_xxs: tuning quantization * iq3_xxs: starting to look better PPL on wiki.test.raw LLaMA-v1-7B: 6.4218 LLaMA-v2-7B: 6.3560 Mistral-7B : 6.0717 This is better than Q3_K_XS, with a 5% reduction in quantized model size. * iq3_xxs: CUDA dot product We have PP-512: 5891 t/s TG-128: 143.9 t/s * iq3_xxs: scalar and AVX2 dot products * iq3_xxs: ARM_NEON and Metal Metal performance is decent, ARM_NEON is pathetic * iq3_xxs: slightly better grid points * Faster iq3_xxs and iq2_xs dot products on CUDA * iq3_xxs: add some quant mix * iq3_xxs: fix failing quantization test Dot product still fails. Is this real? * iq3_xxs: hopefully fix ROCm * iq3_xxs: failing tests This time the dot product accuracy did find an actual bug in the AVX2 implementation. * Add IQ3_XXS to test-backend-ops --------- Co-authored-by: Iwan Kawrakow…4649943
-
ggml alloc: Fix for null dereference on alloc failure (llama/5200)
Paul TsochantarisUnverified signature* Fix for a null pointer dereference if a metal GGML buffer fails to be allocated * Freeing the allocated buffers rather than the pointer in ggml-alloc.c * Fixed the fix of the fix8181686 -
Nomic Vulkan backend (llama/4456)
Unverified signatureSigned-off-by: Jared Van Bortel <jared@nomic.ai> Co-authored-by: niansa <anton-sa@web.de> Co-authored-by: Adam Treat <treat.adam@gmail.com> Co-authored-by: Aaron Miller <apage43@ninjawhale.com> Co-authored-by: ToKiNoBug <tokinobug@163.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: slaren <slarengh@gmail.com>f5fd92d -
ggml : add max buffer sizes to opencl and metal backends (llama/5181)
slarenUnverified signature3d354d0
-
metal : free metal objects (llama/5161)
Paul TsochantarisUnverified signature* Releasing MTLFunction references after Metal pipeline construction * Keeping the `ggml_metal_kernel` structure * Spacing fix * Whitespace fixea7167a
-
gguf : fix comparison (ggml/715)
Unverified signatureggml-ci80cfca4 -
`ggml_cuda_cpy` support for 4d tensors and float16->float32 upcasting (ggml/686)
John Balis +1: slarenUnverified signature* added cuda float16->float32 upcasting to ggml_cuda_cpy * added ability to copy 4d tensors with the cuda backend * added tests for float16_>float32 upcast and 4d tensor cuda copys * added 4d copy test for float32->float16 copy * applied patch suggested by @iamlemec * simplify cpy tests --------- Co-authored-by: slaren <slarengh@gmail.com>75d438c -
gguf : add input validation, prevent integer overflows (ggml/709)
Unverified signature* gguf : add input validation, prevent integer overflows ggml-ci * gguf : fix switch default case * gguf : sanitize info->n_dims and info->type ggml-ci * gguf : assert GGUF_TYPE_SIZE access ggml-ci * ggml : assert mallocs are successful ggml-ci * gguf : prevent integer overflow * gguf : sanitize tensor info ggml-ci * gguf : stricter limit on the number of items ggml-ci5bf1614 -
ci : fix yolo URLs + fix metal capture (ggml/712)
Unverified signature588f789 -
metal : add debug capture backend function (ggml/694)
Jack Mousseau +1: ggerganov Unverified signatureCo-authored-by: Georgi Gerganov <ggerganov@gmail.com>ece88c3
-
common : fix wav buffer detection (#1819)
Unverified signaturebc84057 -
server : add fields to `verbose_json` response (#1802)
Unverified signature* server: include additional fields in the verbose_json response as OpenAI does * server: show request examples on home page * server: todo note for compression_ratio and no_speech_prob * server: add simple demo form to the homepage763d09d -
make : update MSYS_NT (#1813)
Unverified signatureI just upgraded the R wrapper at https://github.com/bnosac/audio.whisper to use whisper.cpp 1.5.4 I'm working on Windows and noticed while doing that that it did not pick up the relevant CFLAGS/CXXFLAGS as my system showed ``` I whisper.cpp build info: I UNAME_S: MSYS_NT-10.0-19045 I UNAME_P: unknown I UNAME_M: x86_64 ``` Many thanks for all the tremendous hard work on maintaining whisper.cpp!587152f
-
talk-llama : sync llama.cpp
Unverified signature1453539 -
sync : ggml
Unverified signature278a9b3 -
ggml : add Vulkan backend (llama/2059)
Unverified signature* Vulkan loader code * Fix matmul kernel, continue implementation * Continue implementation * Vulkan memory management * Vulkan development * Matmul call * Add aligned malloc and free for VMA * Continue implementation * First matmul success * GEMM Kernel optimization * 1D Blocktiling * 2D Blocktiling * Write coalescing * Continue vulkan implementation and optimization * First FP16 attempt, disabled for now * Code abstraction, FP16 implementation, fix kernel, add FP16 to FP32 kernel * Enable device extensions properly, restore fp16 matmul op * Fix mulmat_f16 * Output FP32 in fp16 matmul shader * Fix f16_to_f32 kernel * dequant_q4_0 kernel * Add VMA library * Avoid requesting dedicated memory, VMA can decide that by itself * Add bounds checking to matmul kernels, improve implementation, fix command buffers not freed properly * add cmake commands * Add 2d write operation, profiling code * Fix 2d write * Fix queue selection for AMD RADV * Fix trailing whitespace…5a97aba -
ggml : add unified SYCL backend for Intel GPUs (llama/2690)
Abhilash Majumder +4: jianyuzh, KevinLy, hengyu, ggerganov Unverified signature* first update for migration * update init_cublas * add debug functio, commit all help code * step 1 * step 2 * step3 add fp16, slower 31->28 * add GGML_LIST_DEVICE function * step 5 format device and print * step6, enhance error check, remove CUDA macro, enhance device id to fix none-zero id issue * support main device is non-zero * step7 add debug for code path, rm log * step 8, rename all macro & func from cuda by sycl * fix error of select non-zero device, format device list * ren ggml-sycl.hpp -> ggml-sycl.h * clear CMAKE to rm unused lib and options * correct queue: rm dtct:get_queue * add print tensor function to debug * fix error: wrong result in 658746bb26702e50f2c59c0e4ada8e9da6010481 * summary dpct definition in one header file to replace folder:dpct * refactor device log * mv dpct definition from folder dpct to ggml-sycl.h * update readme, refactor build script * fix build with sycl * set nthread=1 when sycl, increase performance * add run script, co…01169e0 -
ggml : minor type fix (int64_t -> size_t)
Unverified signature1bbb1a9
-
common : fix input buffer check (#1812)
Unverified signature6c38a7f -
talk-llama : sync llama.cpp
Unverified signature92cfd93 -
sync : ggml
Unverified signature5a9540e
-
Add OpenCL add kernel (llama/5151)
Unverified signature* Add OpenCL add kernel * Put add kernel into different string to stay within MSVC string length limit, disable float16 support due to bad resultsf833987 -
cuda : fix tensor size calculation for non-split buffer (llama/5145)
slarenUnverified signature8f3eb65 -
ggml-alloc : add 10% margin to the buffer sizes (llama/5149)
slarenUnverified signaturec55bdf8 -
ggml : update softmax n_task calculation (llama/5126)
snadampalUnverified signatureupdated the n_task calculation to use max number of threads possible. This has improved the prompt eval performance by around 5% for DOT kernels and by around 10% for MMLA kernels on AWS Graviton3.3a3eb8e -
metal : remove unused `n_buffers` and `buffers` (llama/5129)
Paul TsochantarisUnverified signaturea3e87d3
-
metal : show compile log messages
Unverified signatureae08f31
-
cuda : fix 2-bit quants on amd hip (llama/5105)
Engininja2Unverified signature* cuda : fix 2-bit quants on amd hip * use __low2float intrinsic function for new quantsaadbd67 -
llama : pre-allocate input tensors in a separate buffer (llama/5100)
slarenUnverified signature20a4ca1
-
metal : disable support for MUL_MAT F32 x F16
Unverified signature7fbc01f -
CUDA: more info when no device code (llama/5088)
Unverified signaturee96ba7d -
minor : clean-up some warnings and style (llama/5094)
Unverified signature* minor : clean-up some warnings and style ggml-ci * ggml : add comment7df090b
-
ggml : parallelize FP32 conversion when using BLAS (llama/5045)
Unverified signature* make GGML_TASK_INIT phase can be run in multithread * multithreaded dequantize in mul_mat when using blas library * minor fixes * update outdated comment * fix coding style * simplify code Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>7bf2c87
