Ed: solved with the help of the async_stream crate.
I’m struggling with the borrow checker!
My problem: I’m using actix-web and rusqlite. I want to return an unlimited number of records from an rusqlite query, and actix provides a Stream trait for that kind of thing. You just impl the trait and return your records from a poll_next() fn.
On the rusqlite side, there’s this query_map that returns an iterator of records from a query. All I have to do is smush these two features together.
So the plan is to put the iterator returned by query_map into a struct that impls Stream. Problem is the lifetime of a var used by query_map. How to make the var have the same lifetime as the iterator??
So here’s the code:
pub struct ZkNoteStream<'a, T> {
rec_iter: Box<dyn Iterator<Item = T> + 'a>,
}
// impl of Stream just calls next() on the iterator. This compiles fine.
impl<'a> Stream for ZkNoteStream<'a, serde_json::Value> {
type Item = serde_json::Value;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
Poll::Ready(self.rec_iter.next())
}
}
// init function to set up the ZkNoteStream.
impl<'a> ZkNoteStream<'a, Result<ZkListNote, rusqlite::Error>> {
pub fn init(
conn: &'a Connection,
user: i64,
search: &ZkNoteSearch,
) -> Result<Self, Box<dyn Error>> {
let (sql, args) = build_sql(&conn, user, search.clone())?;
let sysid = user_id(&conn, "system")?;
let mut pstmt = conn.prepare(sql.as_str())?;
// Here's the problem! Borrowing pstmt.
let rec_iter = pstmt.query_map(rusqlite::params_from_iter(args.iter()), move |row| {
let id = row.get(0)?;
let sysids = get_sysids(&conn, sysid, id)?;
Ok(ZkListNote {
id: id,
title: row.get(1)?,
is_file: {
let wat: Option<i64> = row.get(2)?;
wat.is_some()
},
user: row.get(3)?,
createdate: row.get(4)?,
changeddate: row.get(5)?,
sysids: sysids,
})
})?;
Ok(ZkNoteStream::<Result<ZkListNote, rusqlite::Error>> {
rec_iter: Box::new(rec_iter),
})
}
}
And here’s the error:
error[E0515]: cannot return value referencing local variable `pstmt`
--> server-lib/src/search.rs:170:5
|
153 | let rec_iter = pstmt.query_map(rusqlite::params_from_iter(args.iter()), move |row| {
| ----- `pstmt` is borrowed here
...
170 | / Ok(ZkNoteStream::<Result<ZkListNote, rusqlite::Error>> {
171 | | rec_iter: Box::new(rec_iter),
172 | | })
| |______^ returns a value referencing data owned by the current function
So basically it boils down to pstmt getting borrowed in the query_map call. It needs to have the same lifetime as the closure. How do I ensure that?
UPDATE!
I sort of solved this part of it, or at least got it to compile. I’ve got a reddit post of this too! Someone there hinted that I should use another struct ‘above’ ZkNoteStream. I’m doing that in the code listing below.
ZnsMaker has an init() fn, then you call make_stream() and it returns a ZkNoteStream. The intent is ZnsMaker should be managed so it lasts as long as the ZkNoteStream needs to last. All this bit compiles, great! But when I go to use it in my actix handler, I get a borrowing problem there instead. So I may have just kicked the can down the road.
This part compiles. Wrong types still, should produce Bytes instead of ZkListNotes.
pub struct ZkNoteStream<'a, T> { rec_iter: Box<dyn Iterator<Item = T> + 'a>, } impl<'a> Stream for ZkNoteStream<'a, Result<ZkListNote, rusqlite::Error>> { type Item = Result<ZkListNote, rusqlite::Error>; fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> { Poll::Ready(self.rec_iter.next()) } } pub struct ZnsMaker<'a> { pstmt: rusqlite::Statement<'a>, sysid: i64, args: Vec<String>, } impl<'a> ZnsMaker<'a> { pub fn init( conn: &'a Connection, user: i64, search: &ZkNoteSearch, ) -> Result<Self, Box<dyn Error>> { let (sql, args) = build_sql(&conn, user, search.clone())?; let sysid = user_id(&conn, "system")?; Ok(ZnsMaker { args: args, sysid: sysid, pstmt: conn.prepare(sql.as_str())?, }) } pub fn make_stream( &'a mut self, conn: &'a Connection, // have to pass the connection in here instead of storing in ZnsMaker, for Reasons. ) -> Result<ZkNoteStream<'a, Result<ZkListNote, rusqlite::Error>>, rusqlite::Error> { let sysid = self.sysid; let rec_iter = self .pstmt .query_map(rusqlite::params_from_iter(self.args.iter()), move |row| { let id = row.get(0)?; let sysids = get_sysids(&conn, sysid, id)?; Ok(ZkListNote { id: id, title: row.get(1)?, is_file: { let wat: Option<i64> = row.get(2)?; wat.is_some() }, user: row.get(3)?, createdate: row.get(4)?, changeddate: row.get(5)?, sysids: sysids, }) })?; Ok(ZkNoteStream::<'a, Result<ZkListNote, rusqlite::Error>> { rec_iter: Box::new(rec_iter), }) } }
Ok and here’s the handler function where I receive a query and make the ZnsMaker. But if I create a ZkNoteStream with it, I get a borrowing error. Maybe it would be ok if I immediately consumed it in an HttpResponse::Ok().streaming(znsstream). Got to fix the types first though.
pub async fn zk_interface_loggedin_streaming( config: &Config, uid: i64, msg: &UserMessage, ) -> Result<HttpResponse, Box<dyn Error>> { match msg.what.as_str() { "searchzknotesstream" => { let msgdata = Option::ok_or(msg.data.as_ref(), "malformed json data")?; let search: ZkNoteSearch = serde_json::from_value(msgdata.clone())?; let conn = sqldata::connection_open(config.orgauth_config.db.as_path())?; let mut znsm = ZnsMaker::init(&conn, uid, &search)?; { // borrowed value of znsm doesn't live long enough! wat do? let znsstream = &znsm.make_stream(&conn)?; } Err("wat".into()) } wat => Err(format!("invalid 'what' code:'{}'", wat).into()), } }
I’m glad you found a workaround. I didn’t want to be defeated by the callback idea that I had yesterday so I worked on it some more and came up with a similar-but-different solution where
ZnsMaker
storespstmt
paired with a closure that borrows it. This is again a simplified version of your code that is self-contained:https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=5bd6fb7c1cbf1c9c44c8f4bdbb1e8074
The closure solution avoids the need to pass
conn
tomake_stream
. I don’t know if it would fix the new borrowing error that you got since I did not reproduce that error. But I think it might.Does
znsstream
need to outliveznsm
? If so I think my solution solves the problem. Doesznsstream
need to outliveconn
? That would be more complicated.Edit: Oh! You don’t need to put both the
pstmt
and a closure inZnsMaker
. Instead you can just store a closure if you move ownership ofpstmt
into the closure. That ensures thatpstmt
lives as long as the function that produces the iterator that you want:https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=491258a7dc9bcad9dab08632d68c026b
Edit: Lol you don’t need
ZnsMaker
after all. See my other top-level comment. I learned some things working on this, so time well spent IMO.