EPFL, ETH Zurich and the Swiss National Supercomputing Centre (CSCS) released Apertus today, Switzerland’s first large-scale, open, multilingual language model — a milestone in generative AI for transparency and diversity.

First “modern and powerful” open source LLM?

Key features

  • Fully open model: open weights + open data + full training details including all data and training recipes
  • Massively Multilingual: 1811 natively supported languages
  • Compliant Apertus is trained while respecting opt-out consent of data owners (even retrospectivey), and avoiding memorization of training data
  • lascapi@jlai.lu
    9·
    2 days ago

    Sounds good!

    Is it the first LLM that is open like that (architecture, model weights, and training data and recipes)?

  • KubeRoot@discuss.tchncs.deEnglish
    162·
    2 days ago

    Apertus was developed with due consideration to Swiss data protection laws, Swiss copyright laws, and the transparency obligations under the EU AI Act. Particular attention has been paid to data integrity and ethical standards: the training corpus builds only on data which is publicly available. It is filtered to respect machine-readable opt-out requests from websites, even retroactively, and to remove personal data, and other undesired content before training begins.

    We probably won’t get better, but sounds like it’s still being trained on scraped data unless you explicitly opt out, including anything that may be getting mirrored by third parties that don’t opt out. Also, they can remove data from the training material retroactively… But presumably won’t be retraining the model from scratch, which means it will still have that in their weights, and the official weights will still have a potential advantage on models trained later on their training data.

    From the license:

    SNAI will regularly provide a file with hash values for download which you can apply as an output filter to your use of our Apertus LLM. The file reflects data protection deletion requests which have been addressed to SNAI as the developer of the Apertus LLM. It allows you to remove Personal Data contained in the model output.

    Oof, so they’re basically passing on data protection deletion requests to the users and telling them all to respectfully account for them.

    They also claim “open data”, but I’m having trouble finding the actual training data, only the “Training data reconstruction scripts”…

    • lime!@feddit.nuEnglish
      13·
      2 days ago

      that’s the problem with deletion requests, the data isn’t in there. it can’t be, from a purely mathematical standpoint. statistically, with the amount of stuff that goes into training, any full work included in an llm is represented by less than one bit. but the model just… remakes sensitive information from scratch. ih reconstructs infringing data based on patterns.

      which of course highlights the big issue with data anonymization: it can’t really be done.

  • snikta@programming.devOP
    19·
    2 days ago

    A fully open-source LLM

    As a fully open language model, Apertus allows researchers, professionals and enthusiasts to build upon the model and adapt it to their specific needs, as well as to inspect any part of the training process. This distinguishes Apertus from models that make only selected components accessible.

    “With this release, we aim to provide a blueprint for how a trustworthy, sovereign, and inclusive AI model can be developed,” says Martin Jaggi, Professor of Machine Learning at EPFL and member of the Steering Committee of the Swiss AI Initiative. The model will be regularly updated by the development team which includes specialized engineers and a large number of researchers from CSCS, ETH Zurich and EPFL.

  • dwt@feddit.orgDeutsch
    11·
    2 days ago

    This begs the question: how Good is it? Did anyone test it yet?

  • hperrin@lemmy.caEnglish
    126·
    2 days ago

    Is this an AI community? I thought this was about software.

      • hperrin@lemmy.caEnglish
        222·
        2 days ago

        Hmm. That’s a shame. I wonder if there’s a community that’s just about open source code, not all of the other things people feel like labeling open source.

        • lime!@feddit.nuEnglish
          21·
          2 days ago

          …what? open source is a standardized term defined by the OSI. it’s a licencing term.

        • Sonalder@lemmy.ml
          7·
          2 days ago

          You should check some FOSS, FLOSS communities. Free(dom)/Libre and Open Source Software is more important than Open Source Software itslef in my honest opinion.

          Open Source can be applied to hardware, medicine, lessons, construction blueprints schematics, not only code.

    • Sonalder@lemmy.ml
      22·
      2 days ago

      Open Source is a way to create but is not limited to only software but many different things. LLMs are software. Most open source LLMs are using Open washing to label themselves as Open Source, however it is not. The importance in Open Source is being able to study how it was made and most of open models have closed training data-sets and training method. Apertus is trully open in the sense that they published Open Data and full training details.

      You have the right to be bother by “AI” but let Open Source enthousiasts being… well, enthousiasts when in a field of Open Washing someone created something trully Open Source to the point of sharing it in an Open Source community on a FOSS plateform.

      Edit : Minor corrections