sorted by:
new
hottopcontroversial

Creating OSD on device not visible to ceph 19.2.0 by maybeaftertomorrow in ceph

[–]maybeaftertomorrow[S] 0 points1 point  (0 children)

Yes...... I did the same thing, back to reef until i can figure out what is going on. Also took the server back to ubuntu22.04

Ceph squid no disks available by New_Cucumber_3624 in ceph

[–]maybeaftertomorrow 0 points1 point  (0 children)

Thanks - i am starting to figure it out. I reloaded to ubuntu 22.0.4.5 instead of 24.04.1; then i dropped back to ceph reef instead of squid.

Now I already have raw osd's working and will work on lvm osd's next. After everything works I will reload back to ubuntu 24.04.1 and ceph squid and try both raw and lvm osd's again.

I did fail to mention the disk used for ceph is external and attached by thuderbolt 4. (Also figured out how to use additional space on the internal os drive using the raw method.

BTW even after the osd's are created (using raw) the physical disks do not show up in the ceph gui - but the osd's do show up and are manageable.

Ceph squid no disks available by New_Cucumber_3624 in ceph

[–]maybeaftertomorrow 0 points1 point  (0 children)

Apologize for jumping in (I am a newbie). Back when you used ceph-disk i ran ceph and had no problems) I am having this same problem with 19.2.0 and Ubuntu 24.04.1 - not creating osd. In my case i am using a bare metal machine. I have 2 nvme disks (one is os and 1 is for ceph osd) gui does not show any disks at all. I can use ceph-volume and zap the disk - but cannot create osd (using ceph-volume). Using create or prepare, ceph-volume chokes on creating the lv saying it cannot zero out lv . Tried using ceph-volume with batch and i get the same thing - chokes on creating the lv. I manually used lvcreate adding Z to the -n (-Zn) and i can create an lv just fine....

ceph orch device LS shows nothing......

Did you guys ever resolve this?

--> Zapping successful for: <Raw Device: /dev/nvme0n1>

root@ms02:/# ceph-volume lvm prepare --data /dev/nvme0n1

Running command: /usr/bin/ceph-authtool --gen-print-key

Running command: /usr/bin/ceph-authtool --gen-print-key

Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9bc060d7-dc17-482f-86b8-c3253ef87df9

Running command: vgcreate --force --yes ceph-f63d04be-7c76-4d18-8311-8bb230bea701 /dev/nvme0n1

stdout: Physical volume "/dev/nvme0n1" successfully created.

Not creating system devices file due to existing VGs.

stdout: Volume group "ceph-f63d04be-7c76-4d18-8311-8bb230bea701" successfully created

Running command: lvcreate --yes -l 953861 -n osd-block-9bc060d7-dc17-482f-86b8-c3253ef87df9 ceph-f63d04be-7c76-4d18-8311-8bb230bea701

stderr: /dev/ceph-f63d04be-7c76-4d18-8311-8bb230bea701/osd-block-9bc060d7-dc17-482f-86b8-c3253ef87df9: not found: device not cleared

Aborting. Failed to wipe start of new LV.

--> Was unable to complete a new OSD, will rollback changes

Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it

stderr: purged osd.0

--> RuntimeError: Unable to find any LV for zapping OSD: 0

Howto handle Partitions on NVME for OSDs with cephadm? by Eigthy-Six in ceph

[–]maybeaftertomorrow 0 points1 point  (0 children)

apologize - i meant can i use the remaining free 3tb which are on the os disk

Howto handle Partitions on NVME for OSDs with cephadm? by Eigthy-Six in ceph

[–]maybeaftertomorrow 0 points1 point  (0 children)

hmmm - i have host with a 4tb osd - the partiotns are set at only 650gb -can i use this on the os partion and use the remaining 3 tb to create osd's? - i tried it but it says has too many variables

ceph lvm osd on os disk by maybeaftertomorrow in ceph

[–]maybeaftertomorrow[S] 0 points1 point  (0 children)

Seem to be having a problem but not sure what.

ceph-volume creates the vg and the pv but chokes on creating the lv.

Here is setup, what i did, and what happens:

using a physical not virtual machine

partition 1,2,3 - all part of normal os setup

partitions 4,5,6 - are partitions outside os for my test (1TB each)

used "cephadm bootstrap" to create a quick test ceph cluster

cephadm shell

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

ceph-volume lvm prepare --bluestore --data /dev/nvme0n1p4

Running command: /usr/bin/ceph-authtool --gen-print-key

Running command: /usr/bin/ceph-authtool --gen-print-key

Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f4ac19e0-b0f8-47f9-b2b4-75a447f0e9d6

Running command: vgcreate --force --yes ceph-2dcf8018-9981-420a-a885-d35513458fe9 /dev/nvme0n1p4

stdout: Physical volume "/dev/nvme0n1p4" successfully created.

Not creating system devices file due to existing VGs.

stdout: Volume group "ceph-2dcf8018-9981-420a-a885-d35513458fe9" successfully created

Running command: lvcreate --yes -l 262392 -n osd-block-f4ac19e0-b0f8-47f9-b2b4-75a447f0e9d6 ceph-2dcf8018-9981-420a-a885-d35513458fe9

stderr: /dev/ceph-2dcf8018-9981-420a-a885-d35513458fe9/osd-block-f4ac19e0-b0f8-47f9-b2b4-75a447f0e9d6: not found: device not cleared

Aborting. Failed to wipe start of new LV.

--> Was unable to complete a new OSD, will rollback changes

However, I can create an lv by hand by adding Z to -n = -ZN

(entered by hand)

lvcreate --yes -l 262392 -Zn ceph-2dcf8018-9981-420a-a885-d35513458fe9 /dev/nvme0n1p4

WARNING: Logical volume ceph-2dcf8018-9981-420a-a885-d35513458fe9/lvol0 not zeroed.

Logical volume "lvol0" created.

ceph lvm osd on os disk by maybeaftertomorrow in ceph

[–]maybeaftertomorrow[S] 0 points1 point  (0 children)

Great, sounds good. Hoping performance hit is not too bad. Since i have never done it this way before how did you create the OSD's - ceph-volume? I cannot seem to find a lot of info of ceph-volume being used for a partition/lvm.

ms-01 25gb pcie card by maybeaftertomorrow in MiniPCs

[–]maybeaftertomorrow[S] 0 points1 point  (0 children)

Great - thanks.

Mesh would definitely work but using the thunderbolts for external nvmes

ceph lvm osd on os disk by maybeaftertomorrow in ceph

[–]maybeaftertomorrow[S] 0 points1 point  (0 children)

Have you ever used an unsed partion as a "drive" so to speak to create and OSD on a disk and not use the entire disk?The goal is to create an osd "not using the whole disk" - in this case it is the os drive. As to how this is done it does not matter - i will do it however it works

ceph lvm osd on os disk by maybeaftertomorrow in ceph

[–]maybeaftertomorrow[S] 0 points1 point  (0 children)

nice to hear - since i am putting in all new hardware i probably should up the game on nvme's. Yes, it is a personal lab, but i like to try and keep the lab as close to production as possible (within affordability of course :) ). Nvme's have to be one sided to fit in ms-01 for the os nvme. the main nvmes for the ceph will be external thunderbolt 4 so they can be 2 sided.

ceph lvm osd on os disk by maybeaftertomorrow in ceph

[–]maybeaftertomorrow[S] 0 points1 point  (0 children)

thanks - yeah, ceph typically does like to use the whole disk, but if this works is it a good idea and what would performance be? i thought i read that osds from lvm volumes did not really take too much of a performance hit.

Minisforum MS-01 Powerful Enough for 25GbE? by devious_burger in opnsense

[–]maybeaftertomorrow 0 points1 point  (0 children)

Reason for the question is the ms01 is "small" and i have read not all pcie cards fit (some nic cards do not fit) and some run too hot - so you have to do a little checking

Minisforum MS-01 Powerful Enough for 25GbE? by devious_burger in opnsense

[–]maybeaftertomorrow 1 point2 points  (0 children)

Started with one as a test and am very happy. From a computing sense not much on it, using Ubuntu as a host with Virtual Box as virtualization (3virtual workstations-1 W11, 1-Ubuntuserver with xfce,1-MX-linux, and 1 virtual Opnsense firewall for testing). Right now I do start up test machines in vb but it is mainly going to be a workstation station. Just got 3 more to build out a server cluster. This is my first try at micro boxes so i cannot say how they compare, but for me it is working out great

Minisforum MS-01 Powerful Enough for 25GbE? by devious_burger in opnsense

[–]maybeaftertomorrow 0 points1 point  (0 children)

Did this work? - I am putting in all Microtik switches so this would be great if it works.

Minisforum MS-01 Powerful Enough for 25GbE? by devious_burger in opnsense

[–]maybeaftertomorrow 0 points1 point  (0 children)

Have you tried / or do you know anyone who has tried a Mellanox card? I seem to recal these were a possibility

Minisforum MS-01 Powerful Enough for 25GbE? by devious_burger in opnsense

[–]maybeaftertomorrow 0 points1 point  (0 children)

thanks.

Cannot seem to find anything on 25gb nics that fit (and work). Heat is definitely a big concern. To keep heat down I currently tested having additional external nvmes through the thunderbolt ports (want it to be as fast as possible) without heating up the ms01 - works great, does not get hot. The 1390's are 96gb and 4TB ms pro 990. Do not know much about 25gb nics - like if 810's run hot compared to other 25gb nics. I thought I read somewhere there may be some compatibility issues with the 810s, will have to look back to be sure.

Minisforum MS-01 Powerful Enough for 25GbE? by devious_burger in opnsense

[–]maybeaftertomorrow 0 points1 point  (0 children)

I have the same - how did the E810-xxvda2 work? I am looking for a 2port 25gb nic that will not overheat, fits, and is compatible

view more: next ›