Aode (He/They)

doing some rust dev

  • 6 Posts
  • 45 Comments
Joined 3Y ago
cake
Cake day: Jun 29, 2020

help-circle
rss

I chose it at the start of the project 🤷


Update on this: I got the feature work done this weekend, so now I’ll be testing it a bunch for upgrades and storage migrations


I am aware of garage, but haven’t tested it yet with pict-rs. It’s a cool project for sure


Yes. It uses sha256 rather than perceptual hashing, but that’s Good Enough™️


I’m not against including an ipfs layer in pict-rs, but the complexity would go way up. Federating an image between lemmy servers would require sending the ipfs uri between servers via activitypub, and then each receiving server sending that uri to pict-rs. pict-rs would then need to decide, on each server, if the ipfs-stored image matches that servers’ image requirements (max filesize, max dimensions, etc), and if it does, then that pict-rs server would request to pin the image. I don’t know exactly how ipfs pinning works, but ideally it would only be stored locally if it isn’t already stored in at least N other locations. If the remote image doesn’t match the local server’s configuration, it could either be rejected or downloaded & processed (resized etc).

Serving ipfs-stored images that aren’t replicated locally might also be slow, but I won’t know for sure unless I actually try building this out.


I can only say “when it’s ready.” I think most of what I want to include in 0.4 is there, but I don’t have a ton of time to work on it currently. I might see if I can get my last feature changes in this weekend, then it will be a matter of ensuring the 0.3 -> 0.4 upgrade is smooth, and that storage migration is solid


pict-rs doesn’t keep track of how often it serves different images, so there’s not a good metric for pruning old images. That said, 0.4 will introduce functionality for cleaning up processed images (e.g. resizes/thumbnails), removing their files & metadata. If they are viewed again, they will be re-generated.

0.4 will also include the ability to scale down images on upload, rather than storing the original resolution. This is not yet implemented, but it’s on my roadmap.

All this said, it is already possible to use pict-rs with object storage (s3-compatible), rather than block storage. That’s a good option if your hosting provider offers it


I've just published the stable version of pict-rs 0.3, and included a short write-up in the release description. Not much has changed since I posted about the betas, but check out the release anyway! - v0.3.0 changes: https://git.asonix.dog/asonix/pict-rs/releases/tag/v0.3.0 - v0.3.1 changes: https://git.asonix.dog/asonix/pict-rs/releases/tag/v0.3.1

what I noticed personally is that my pict-rs server had eaten 1,200MB of RAM, which is the limit I allowed that container. it was still working and it automatically restarted on crash so I didn’t notice it oops

after I fixed the problem, my pict-rs server has been running at about 29MB for the last 12 hours. Significant improvement I’d say



Problem:

I gave tokio-console a buffer way too big

Fix:

Update pict-rs to at least v0.3.0-rc.7, which doesn’t enable console by default


how long does it take for pict-rs to use up this memory?


Hey all! Just wanted to share a little fun side project I've been working on recently. It's an ecosystem similar to smol for writing async rust code, but the internals make no use of `unsafe` at all ### repos: - `async-join`: a safe `join_all` function for `Vec<Future>` - `polldance`: a safe polling library built on unix `poll` with `rustix` for memory & io safety - `foxtrot`: a safe async reactor bulit on `polldance` and `rustix` - `jitterbug`: a safe async executor with multi-threaded work-stealing support I'm not here to say "you should definitely use this" because I've wirtten zero tests, and I've made no effort to implement standard apis like AsyncRead, AsyncWrite, etc, but I wanted to show it is possible to build these things yourself Shoutout to [rustix](https://github.com/bytecodealliance/rustix) for making socket programming safe and easy


If you don’t know what OpenTelemetry is but you’re having OpenTelemetry errors in your logs, it likely means that you have an opentelemetry_url set in your config when you shouldn’t. You can get rid of the error by deleting that line, which will disable opentelemetry exports


Lemmy and pict-rs support optionally exporting opentelemetry spans, and setting up Jaeger to capture those is a nice way to manage traces without shipping data to third parties


Plugin is likely lewis6991/gitsigns.nvim



i might try

let res = collection.into_iter().map(|item| item.fallible_operation()).fold(Ok(Vec::new()), |acc, res| {
    match (acc, res) {
        (Ok(mut vec), Ok(item)) => {
            vec.push(item);
            Ok(vec)
        }
        (Err(mut vec), Err(error)) => {
            vec.push(error);
            Err(vec)
        }
        (Ok(_), Err(error)) => Err(vec![error]),
        (acc, _) => acc,
    }
});

Maybe expand it with

pub trait Evert<T, E> {
    fn evert(self) -> Result<Vec<T>, Vec<E>>;
}

impl<I, T, E> Evert<T, E> for I
where
    I: IntoIterator<Item = Result<T, E>>,
{
    fn evert(self) -> Result<Vec<T>, Vec<E>> {
        self.into_iter().fold(/* implementation from above */)
    }
}

fn main() {
    let result = vec![Ok(1), Err(3), Ok(4), Err(8)].evert();
    assert_eq!(result, Err(vec![3, 8]));
}


With these examples I think the correlation is a little loose. Dr Lutchmedial died two weeks after his third shot. Claiming the vaccine was a direct cause when the only available evidence is a two week timespan is weak.

The other example provided doesn’t demonstrate harm caused by a vaccine. It sucks that a deadly virus is deadly, and it sucks that the vaccine didn’t help in this case. It’s not a good argument against getting a vaccine.


I’m a sponsor on github for spacejam (github user) and elementary OS, and I’m a patron of Hector Martin on patreon.


First, it doesn’t take an array as input, it takes a slice as input. You can turn a Vec<&str> into a slice &[&str] by borrowing it.

Second, the human-sort crate’s api is bad, because rarely to people have Vec<&str> or &[str] and there’s no way to produce a Vec<&str> from a Vec<String> without creating a new vector that borrows from the first.

let mut strs = Vec::new();

for s in strings.iter() {
    strs.push(s);
}

human_sort::sort(&mut strs);

What human-sort should have done was accept &[T] where T: Deref<Target = str> which would allow passing a Vec of String or a Vec of &str.

Feel free to open a PR against human-sort, but it looks like it hasn’t been updated in a couple years so it might not be maintained anymore


Pawoo is a mastodon server run by the same company that owns pixiv. They’re using the default mastodon description but have also deliberately added tracking because they’re a company that doesn’t care about privacy.

I dont really know what point this post is trying to make. There are other mastodon servers you can join that wont track your activity.


specifically my pict-rs development looks like:

$ sudo docker run --rm -it -p 8080:8080 -v "$(pwd):/mnt" archlinux:latest /bin/sh
# pacman -Syu --noconfirm imagemagick perl-image-exiftool ffmpeg
# PATH=$PATH:/usr/bin/vendor_perl ./mnt/target/debug/pict-rs -p /mnt/data

:D I kinda do this when working on pict-rs. I have a container with the proper runtime dependencies, and then I cargo build on my host

I definitely recommend it over re-building a container for each change


It also has .flush() and .flush_async() to flush inline with your application if you need that


Sled is my go-to key value store in Rust. It flushes to disk every 500ms by default iirc


I love writing rust and I’ve done all my personal projects for the last 4 or 5 years in it, but I dont think it’s a good idea for other existing languages to adopt more of Rust’s features.

Languages like Elixir or Lisp are so different it doesn’t make any sense to adopt Rust semantics or syntax, and anything with a garbage collector, Go included, doesn’t need to care about ownership, borrowing, or lifetimes (besides the usual passing references being cheaper than passing whole objects).

Languages like C or C++ can definitely be replaced in many cases with Rust, but I dont think making them more like rust is a wise decision. Adopting something like a borrow checker in c or c++ completely changes the language in a way thats not only backwards incompatible, but also probably not welcome by developers already working in those languages.

All this to say: I’d prefer more people to work in rust than to alter other languages to be more like rust, but I also think there are a number of other languages worth learning and working in. To a large extent, choosing a language is about taste.


oh I forgot to mention pict-rs is now nicely instrumented with the tracing framework and can export spans to OTLP collectors if the opentelemetry_url configuration option is set. It supports reading OpenTelemetry span information out of HTTP requests to enable tracing requests through the calling application to pict-rs and back. This is incredibly helpful for debugging.

Since lemmy is not instrumented (yet) with tracing, the benefit to lemmy admins and developers is not much, but this prepares for a future when lemmy is instrumented.

This also changes how pict-rs logs look. Hopefully admins and developers can glean more information from pict-rs logging now, especially in the event of an error.


pict-rs 0.3 betas rolling out
Hey all! I'm the developer of pict-rs, the image host API used by lemmy. I'm here to officially announce the availability of the pict-rs 0.3 beta releases. As I type this, v0.3.0-beta.3 is currently building and should be published to dockerhub shortly. Here's what's new # Api There are now APIs to fetch metadata about images. These endpoints are mentioned in pict-rs's README.md, but at a high level they can return mime-type, dimensions, and creation date for original or processed images The rest of the pict-rs APIs have not changed, so if you experience different behavior, please let me know in the [#pict-rs:matrix.asonix.dog](https://matrix.to/#/#pictrs:matrix.asonix.dog) channel on matrix. # Dependencies pict-rs no longer links against imagemagick, ffmpeg, and gexiv2. This change makes developing on pict-rs significantly easier, since it now compiles like a normal rust application with very few system dependencies. This also means it can be easily linked against the musl standard library to create a static executable. Instead of linking against the programs mentioned above, pict-rs will now spawn imagemagick, ffmpeg, and perl-image-exiftool processes in order to manipulate images. The required versions are - Imagemagick: At least 7.0 - ffmpeg: At least 4.0 - perl-image-exiftool: whatever Arch or Alpine are packaging right now # Files images are now organized into directories by default, and will no longer have original files dumped into the root of `files/`. If upgrading from 0.2, a migration script will attempt to move existing files from their current locations to this new structure. Please please please take a backup of your pict-rs storage directories before upgrading. # Configuration pict-rs now supports configuring the service with a file, rather than relying on commandline arguments or environment variables. The majority of settings can still be manipulated through the environment or through the commandline, but there is a specific feature (mentioned below) that can only be accessed through a configuration file. For information on configuring pict-rs, take a look at the [pict-rs.toml file](https://git.asonix.dog/asonix/pict-rs/src/branch/main/pict-rs.toml) # Object Storage There is now object storage support built-in to pict-rs. If you already have a pict-rs deployment as part of your lemmy server, there is an included migration feature to help move from your existing block storage to object storage for the image files. This does not completely remove pict-rs dependency on a local filesystem, but all original images and transformed images will be stored in object storage rather than on the local filesystem. In particular, pict-rs database still resides on the local filesystem, and temporary files created when manipulating images will still use the /tmp directory. Object Storage has not been tested with any cloud provider, but I have set up a local [minio](https://min.io/) container for local testing. If you try to use pict-rs with your favorite cloud provider and it doesn't work, please let me know in the [#pict-rs:matrix.asonix.dog](https://matrix.to/#/#pictrs:matrix.asonix.dog) channel on matrix. If you are migrating from local storage to object storage, the specific way to accomplish that is with pict-rs's `--migrate-file <file>` flag, and the associated migration file. An example can be found [here](https://git.asonix.dog/asonix/pict-rs/src/branch/main/migrate.toml) Finally, object-storage is behind a feature flag, and can be disabled in custom builds of pict-rs by providing the `--no-default-features` flag to cargo when building # io-uring Since actix-rt gained support for io-uring with 2.3.0, I have implemented an io-uring file backend for pict-rs. It is disabled by default, as io-uring requires a recent linux kernel release and I don't want to publish an application that doesn't work on most people's computers, but if you are interested in building and running an io-uring version of the application, you can pass `--features io-uring` to cargo to produce an io-uring backed binary rather than traditional blocking file IO. In my personal testing, io-uring performed significantly better for serving files on my laptop running linux kernel 5.13. Keep in mind that io-uring in the actix-web ecosystem is currently considered unstable and is therefore exempt from semver, so the io-uring feature in pict-rs should also be considered unstable. # Links That about wraps up this announcement! Here's some useful links for pict-rs: - git: https://git.asonix.dog/asonix/pict-rs - matrix: https://matrix.to/#/#pictrs:matrix.asonix.dog - dockerhub: https://hub.docker.com/r/asonix/pictrs - asonix on mastodon https://masto.asonix.dog/@asonix

As an update, I’m back online on cable internet instead of fiber. I still need to get our fiber fixed but paying a different provider to get us online in the meantime made some amount of sense.

git.asonix.dog should work now



If you want all the code required to build pict-rs I can try getting it all working on yerbamate.dev this evening


Yup, my internet is down. A fiber line in our neighbor’s attic was broken (assuming an animal chewed through it). We’re trying to get back online but it’s taking some time. So far we’ve been offline for around 87 hours :/


There’s no date on the article posted, but the one it references is from February 2018


I love the rock64s! I have a number that I do self-hosting on :3

Real excited for the Quartz64 model B too 👀


For those who don’t know, it’s Rust

I also do a lot of JS and Ruby at work, though



also, since you’re running this on elementary OS, I’m assuming that’s your personal computer. are you able to access jellyfin from other computers on your local network?


I have a few questions, being familiar with self hosting but not with jellyfin specifically

what ports does jellyfin expose? what IP does it have? what ports did you forward? how did you forward them? what IP did you forward them to? how are you telling your friends to connect?


I went through this process the other day on my pinebook pro with manjaro, a few tweaks were required to get it working, but I’m happy with the result!


Some general notes: I feel like blocking operations are faster now with tokio’s spawn_blocking underneath than before when it was actix-threadpool. I don’t have any benchmarks to demonstrate this, but I think I noticed an improvement when uploading files with actix-form-data + actix-fs in pict-rs while testing the upgrade.


I just upgraded a bunch of stuff to actix-web 4.0.0-beta.3
And it went really well. So far the only changes I noticed were the `web::block` error api (due the the use of tokio's spawn_blocking now instead of actix-threadpool), the move of Service's Request from being an associated type to a generic, the move to `insert_header` and `append_header`, and the inclusion of the correct `JsonPayloadError` so I don't need to depend on awc to get that type :) Here's what I've gotten done - [background-jobs-actix](https://git.asonix.dog/asonix/background-jobs) updated to actix-rt 2.0.2 - [background-jobs-sled-storage](https://git.asonix.dog/asonix/background-jobs) revived on actix-rt 2.0.2 - [actix-fs (unpublished)](https://git.asonix.dog/asonix/actix-fs) updated to actix-rt 2.0.2 - [actix-webfinger](https://git.asonix.dog/asonix/actix-webfinger) updated to actix-web 4.0.0-beta.3 - [actix-form-data](https://git.asonix.dog/asonix/actix-form-data) updated to actix-web 4.0.0-beta.3 - [http-signature-normalization-actix](https://git.asonix.dog/asonix/http-signature-normalization) updated to actix-web 4.0.0-beta.3 - [pict-rs](https://git.asonix.dog/asonix/pict-rs) updated to actix-web 4.0.0-beta.3 - [pict-rs-proxy](https://git.asonix.dog/asonix/pict-rs-proxy) updated to actix-web 4.0.0-beta.3 - [pict-rs-aggregator](https://git.asonix.dog/asonix/pict-rs-aggregator) updated to actix-web 4.0.0-beta.3 - [aode-relay](https://git.asonix.dog/asonix/relay) updated to actix-web 4.0.0-beta.3 My current Big Project, Hyaenidae, has not been updated because actix-session doesn't yet have a beta release, but I'm keeping my eye out. One question I have about the Service API changing is: why. I'm interested to know if there's a specific benefit for moving Request to be a generic instead of an associated type. I don't deal with Service much outside of writing a few middlewares, so this change doesn't make a ton of sense to me.

Oops! I lost my postgres
I'm writing this to hopefully get the attention of [@dessalines@lemmy.ml](https://lemmy.ml/u/dessalines) or [@nutomic@lemmy.ml](https://lemmy.ml/u/nutomic) to let them know that I lost my database today, which means my gitea (where pict-rs is hosted) is down until I get a new database in place and upload the source again I think I'll have that up again this evening In related news I will not be on mastodon or matrix for a while

Anyone use Sway on a pinebook pro?
I've had mixed results trying to use Sway on my pinebook pro. Before I reflashed manjaro recently, Sway seemed to run alright, but now I get a perpetual black screen when trying to log into Sway. If I launch it from within my MATE environment it works fine.