Cannot ssh into the container when created using podman with proper ssh-keys as well

I am trying to create a ubuntu container using podman and able to bring up the container successfully. However when I try to ssh into the container using ssh-keys it is not working. Here is my Dockerfile looks like

FROM ubuntu:latest
RUN apt update && apt install  openssh-server sudo -y
# Create a user “sshuser” and group “sshgroup”
RUN groupadd sshgroup && useradd -ms /bin/bash -g sshgroup sshuser
# Create sshuser directory in home
RUN mkdir -p /home/sshuser/.ssh
# Copy the ssh public key in the authorized_keys file. The idkey.pub below is a public key file you get from ssh-keygen. They are under ~/.ssh directory by default.
COPY id_rsa.pub /home/sshuser/.ssh/authorized_keys
# change ownership of the key file. 
RUN chown sshuser:sshgroup /home/sshuser/.ssh/authorized_keys && chmod 600 /home/sshuser/.ssh/authorized_keys
# Start SSH service
RUN service ssh start
# Expose docker port 22
EXPOSE 22
CMD ["/usr/sbin/sshd","-D"]

When I tried ssh into the container using private key I am not getting any error and ssh is not successful

# ssh -i id_rsa [email protected]
Connection to 10.88.0.17 closed by remote host.
Connection to 10.88.0.17 closed.

Could some one let me know what could be the issue

Here are the logs in ssh -v mode

# ssh -v -i id_rsa [email protected]
OpenSSH_8.9p1 Ubuntu-3ubuntu0.7, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 10.88.0.17 [10.88.0.17] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.6p1 Ubuntu-3ubuntu13
debug1: compat_banner: match: OpenSSH_9.6p1 Ubuntu-3ubuntu13 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.88.0.17:22 as 'sshuser'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:FXSPxYM08uFYOjR74E0eIE/UkwxP/PUU0IJ2+7gjy0I
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '10.88.0.17' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:3
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /root/.ssh/id_rsa RSA SHA256:tkHV+TNAKpWzAo169/l7YxWjLxHj7uQJ/ibr7IrVjW0 explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256>
debug1: kex_input_ext_info: [email protected]=<0>
debug1: kex_input_ext_info: [email protected] (unrecognised)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsaRSA SHA256:tkHV+TNAKpWzAo169/l7YxWjLxHj7uQJ/ibr7IrVjW0 explicit
debug1: Server accepts key: /root/.ssh/id_rsa RSA SHA256:tkHV+TNAKpWzAo169/l7YxWjLxHj7uQJ/ibr7IrVjW0 explicit
Authenticated to 10.88.0.17 ([10.88.0.17]:22) using "publickey".
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts for 10.88.0.17 / (none)
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts2 for 10.88.0.17 / (none)
debug1: client_input_hostkeys: hostkeys file /root/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: no new or deprecated keys from server
debug1: Remote: /home/sshuser/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/sshuser/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_GB.UTF-8"
debug1: channel 0: free: client-session, nchannels 1
Connection to 10.88.0.17 closed by remote host.
Connection to 10.88.0.17 closed.
Transferred: sent 4056, received 3148 bytes, in 0.1 seconds
Bytes per second: sent 64566.0, received 50111.9
debug1: Exit status -1